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PREFACE 


This  report  has  been  prepared  by  the  Computer  Systems  Laboratory  (formerly  the  Institute  for 
Computer  Sciences  and  Technology).  It  is  the  fifth  report  issued  in  this  area.  The  first  report 
was  a  summary  of  the  Application  Development  Productivity  Workshops,  held  on  November  IS- 
IS, 1985  at  the  National  Institute  of  Standards  and  Technology  (formerly  the  National  Bureau  of 
Standards.)  Much  of  the  discussion  at  this  workshop  was  an  urgent  requirement  for  information 
and  guidance  on  4GLs.  This  impems  set  the  stage  for  accelerating  research  and  the  dissemination 
of  information  in  this  area. 

The  second  report  was  A  Functional  Model  for  Fourth  Generation  Languages,  NBS  Special 
Pubhcation  500-138,  by  Gary  E.  Fisher  [FISH86],  issued  in  June  1986.  The  purpose  of  this 
functional  model  was  to  define  4GLs  "in  a  manner  similar  to  specifying  the  functions  of  a  specific 
software  appUcation."  This  definition  allowed  managers,  technical  personnel,  and  end-users  to  refer 
to  commonly  understood  terminology  in  the  4GL  context.  In  addition,  it  defined  the  interfaces 
between  4GLs,  humans,  operating  systems,  peripheral  devices,  and  other  appUcation  software. 

The  third  report  was  Guide  to  the  Selection  and  Use  of  Fourth  Generation  Languages,  NBS 
Special  PubUcation  500-143,  by  Martha  Mulford  Gray  [GRAY86],  issued  in  September  1986.  The 
report  provided  guidance  on  the  selection  process  in  general  for  4GLs;  a  description  of  the  features, 
functions,  and  capabihties  of  4GLs;  and  a  brief  discussion  of  the  use  of  4GLs.  "Functional 
Benchmarks  for  Fourth  Generation  Languages"  should  be  used  with  the  guidance  provided  in 
[GRAY86]. 

The  fourth  report  was  Application  Software  Prototyping  and  Fourth  Generation  Languages, 
NBS  Special  PubUcation  500-148,  by  Gary  E.  Fisher  [nSH87],  issued  May  1987.  This  report  was 
designed  as  an  introduction  to  the  planning,  organizing,  executing,  and  controUing  of  a  methodology 
for  appUcation  prototyping.  The  report  recommended  that  since  4GLs  provide  many  of  the 
capabiUties  necessary  for  prototype  development,  a  combination  of  appUcation  prototyping  and  4GLs 
can  provide  a  cost-effective  and  controUable  method  of  developing  and  maintaining  software. 

This  fifth  report  represents  years  of  research  and  the  efforts  of  not  only  the  authors  but  also  of 
numerous  stalT  and  students  who  installed  and  tested  ten  4GL  products,  executed  and  refined  the 
benchmarking  tasks,  and  assisted  in  the  analyses  of  the  results.  We  would  especiaUy  like  to 
acknowledge  the  efforts  of  Laura  Benedict,  Jon  Berry,  Dawn  HUl,  Nadine  Melinger,  Dan  Nielsen, 
Teresa  Stansbury,  and  Doug  White. 
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ABSTRACT 


In  recent  years,  fourth  generation  language  (4GL)  usage  has  expanded  in  data  processing 
organizations,  especially  where  end-users  have  assumed  increased  programming  responsibilities. 
Since  there  are  no  4GL  standards,  managers  selecting  a  4GL  need  some  method  of  determining  how 
well  a  particular  4GL  will  meet  organizational,  application,  and  user  requirements.  This  report 
provides  a  methodology  to  assist  in  that  determination.  It  contains  "functional  benchmarks"  (as 
opposed  to  performance  benchmarks)  consisting  of  a  testing  methodology  and  descriptions  of  tests 
to  evaluate  the  capabilities  of  a  particular  4GL  in  relation  to  organizational  requirements.  Test 
results  are  evaluated  at  two  levels^:  the  4GL's  ability  to  perform  a  task,  and  the  ease  of  performing 
it.  These  evaluations  are  combined  with  user-defined  weighted  requirements  to  produce  an  overall 
rating  for  each  4GL  tested. 


1.  INTRODUCTION 

The  label  "Fourth  Generation  Language"  (4GL)  has  been  applied  to  a  wide  variety  of  products. 
This  variety  makes  any  attempt  to  create  generic  evaluation  measures  difficult.  Because  each  4GL 
product  is  different,  the  most  important  measure  is  the  product's  abiUty  to  fulfill  the  requirements 
of  the  appUcation  at  hand.  This  ability  cannot  always  be  determined  from  a  vendor's  demonstration 
or  another  user's  opinion.  The  best  method  of  evaluation  is  to  test  how  well  the  4GL  satisfies  the 
requirements  of  the  application  and  to  record  the  results. 

NBS  Special  Publication  500-143,  Guide  to  the  Selection  and  Use  of  Fourth  Generation  Languages, 
provides  guidance  on  the  selection  process  for  4GLs;  a  description  of  the  features,  functions,  and 
capabilities  of  4GLs;  and  a  brief  discussion  of  the  use  of  4GLs.  One  of  the  steps  (step  nine)  in 
the  selection  process  described  in  that  report  is  to  "analyze  the  top  few  choices  in  detail  using  a 
benchmark,  pilot  test,  demonstrations  and/or  user  comments."  This  report  describes  functional 
benchmarks  that  could  be  used  for  this  step.  It  contains  a  testing  methodology  and  a  set  of 
functional  benchmark  tasks  to  assist  managers  in  determining  if  a  proposed  4GL  meets  an 
appUcation' s  requirements. 

1.1  Approach 

The  term  "functional  benchmark"  was  derived  from  the  Federal  Information  Processing  Standard, 
Guidelines  for  Benchmarking  ADP  Systems  in  the  Competitive  Procurement  Environment,  FIPS 
PUB  42-1. 

"For  these  guidelines  the  term  'benchmarking'  is  used  to  convey  the  same  meaning  as  the  more 
explicit  term  'benchmark  mix  demonstration.'  A  'benchmark  mix  demonstration,'  sometimes  referred 
to  as  a  Live  Test  Demonstration  (LTD),  consists  of  a  user-witnessed  running  of  a  group  (mix)  of 
programs  representative  of  the  user's  predicted  workload  on  a  vendor's  proposed  computer  system 
in  order  to  validate  system  performance.  Another  type  of  demonstration  that  is  frequently  called 
'benchmarking/  more  properly  should  be  referred  to  as  either  a  capability  demonstration  or  a 
functional  demonstration.  The  latter  type  of  demonstration  is  intended  to  show  only  system  or 
functional  capabilities  in  some  specific  areas  without  regard  to  total  system  performance." 
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The  methodology  presented  in  this  report  is  based  on  a  study  of  fourth  generation  languages 
performed  by  the  Computer  Systems  Laboratory  (CSL),  of  the  National  Institute  of  Standards  and 
Technology  (NIST),  formerly  the  National  Bureau  of  Standards  (NBS).  To  date,  this  research  has 
generated  a  workshop  summary  and  three  publications:  A  Functional  Model  for  Fourth  Generation 
Languages,  NBS  Special  Publication  500-138  [FISH86],  Guide  to  the  Selection  and  Use  of  Fourth 
Generation  Languages,  NBS  Special  Publication  500-143  [GRAY86],  and  Application  Software 
Prototyping  and  Fourth  Generation  Languages,  NBS  Special  Publication  500-148  [FISH87]. 

As  part  of  this  4GL  research,  test  databases  were  created  and  about  60  tasks  developed  and 
executed  on  various  4GL  products  to  evaluate  and  measure  characteristics  of  4GLs.  These  specific 
tasks  were  modified  and  augmented  to  accommodate  the  diverse  functionality  provided  by  different 
4GLs.  A  generalized  benchmarking  methodology  for  measuring  the  functionality  (not  performance) 
of  4GLs  was  developed  from  experience  with  these  tasks  and  4GL  products. 

The  generalized  tasks  were  rearranged  and  respecified  to  remove  as  much  bias  and  implementation 
difficulty  as  possible.  The  improved  set  of  tasks  was  then  reordered  in  a  manner  that  generally 
would  require  the  least  amount  of  work  in  performing  tests  of  a  specific  4GL.  Most  of  the  basic 
tasks  were  tested  on  microcomputer  versions  of  the  4GL  products  but  the  generalized  methodology 
and  tasks  should  be  appUcable  to  any  hardware  that  will  support  the  4GL  products. 

There  are  over  100  requirements  and  over  200  tasks  included  in  this  report.  The  time  required  to 
complete  all  of  the  tasks,  if  all  of  the  requirements  were  needed,  is  difficult  to  estimate  because 
it  is  dependent  on  the  knowledge  and  expertise  of  the  person  mnning  the  tasks.  If  the  person  is 
familiar  with  the  particular  4GL  that  is  being  tested,  completing  the  tasks  will  require  much  less 
time  than  for  someone  with  no  4GL  experience  because  the  testing  time  has  to  include  the  4GL 
leaming  curve.  We  were  able  to  complete  all  of  the  tasks  in  a  few  days  for  some  of  the  products 
and  a  few  weeks  for  others.  Users  with  no  4GL  experience  might  require  twice  as  much  time. 

Since  functional  benchmarking  is  only  part  of  the  4GL  selection  and  evaluation  process,  the 
methodology  described  in  this  report  will  be  most  useful  if  used  in  conjunction  with  the  selection 
process  described  in  [GRAY86].  The  selection  process  is  summarized  in  the  following  section. 

1.2  4GL  Selection  Process 

Ten  steps  in  the  4GL  selection  process  are  defined  in  [GRAY86]  and  illustrated  in  figure  1.  The 
following  presents  a  summarized  version  of  the  process  and  provides  guidance  on  the  selection 
process  as  a  whole.  The  ten-step  process  should  be  completed  whether  you  plan  on  purchasing  a 
large,  medium,  or  small  system.  The  amount  of  depth  and  detail  in  each  step  will  change 
depending  on  the  size  of  the  procurement. 

The  selection  process  begins  with  step  one,  a  brief  description  of  the  problems  that  are  to  be 
solved  or  the  applications  that  are  to  be  implemented  in  the  4GL. 

The  second  step  is  to  complete  an  analysis  of  the  application  environments,  i.e.,  the  hardware, 
software,  organizational,  and  user  environments. 

The  third  step  is  to  decide  how  the  selection  decision  will  be  made,  i.e.,  if  there  will  be  a 
selection  and  evaluation  team  or  a  single  decision-maker. 
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Figure  1.   4GL  Selection  Process 


The  fourth  step  is  to  define  the  requirements  for  the  4GL  based  on  the  information  gathered  in 
the  first  two  steps.  Emphasis  must  be  placed  here  on  defining  actual  requirements,  and  not  on 
unrealistic  demands,  wishes,  etc. 

After  the  requirements  have  been  defined,  a  list  of  4GL  features  which  can  fulfill  these 
requirements  must  be  developed.  (See  [GRAY86]  for  a  full  description  of  these  features.)  This 
is  step  five  of  the  selection  process. 

Step  six  is  to  develop  a  ranking  or  rating  system  for  the  4GL  features  in  order  to  differentiate 
between  the  most  desired  and  the  least  desired  features. 
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Step  seven  consists  of  selecting  candidate  packages.  Information  on  the  availability  of  4GLs  can 
be  obtained  from  computer  literature,  software  reference  services,  trade  shows,  computer 
conferences,  and  computer  user  groups. 

Step  eight  is  to  eliminate  most  of  the  candidate  packages.  First,  a  screening  process  should  be 
established  to  eliminate  packages  that  do  not  fulfill  the  hardware,  operating  system,  or  mandatory 
feature  requirements,  or  that  possess  undesirable  qualities.  After  the  screening  process,  further 
elimination  of  products  should  be  based  on  rating  the  other  feamres  of  the  4GLs  and  comparing 
these  ratings  witli  the  user  ratings  of  the  desired  features.  This  step  should  eliminate  all  but  a  few 
products. 

Step  nine  is  to  analyze  the  top  few  candidate  products  in  detail.  Approaches  such  as 
benchmarking,  pilot  testing,  vendor  demonstrations,  and  gathering  user  opinions  can  be  used  for 

this  step. 

The  final  step,  step  ten,  is  to  select  the  most  appropriate  package. 
2.  FUNCTIONAL  BENCHMARKING  METHODOLOGY 

The  functional  benchmarking  methodology  is  designed  to  determine  if  a  4GL  has  the  functionahty 
necessary  to  meet  the  requirements  of  an  appUcation.  The  requirements  are  dependent  on  what 
needs  to  be  done  and  by  whom;  specifically,  the  characteristics  of  the  application,  and  the  charac- 
teristics of  the  users  who  will  be  working  with  these  appUcations.  These  requirements  must  be 
determined  before  the  functional  benchmark  tasks  can  be  performed. 

^  The  second  step  of  the  selection  process  in  [GRAY86]  is  to  analyze  the  application  environment. 
A  process  is  provided  in  [GRAY86]  for  analyzing  this  appUcation  environment,  which  includes  the 
hardware,  software,  organizational,  and  user  environments.  The  results  of  this  analysis  are  used  in 
step  four  to  define  the  application  requirements.  For  example,  a  typical  requirement  in  data 
processing  environments  is  die  need  for  the  4GL  to  provide  programming  interfaces  for  both  casual 
and  expert  users.  For  the  casual  user,  a  menu-oriented  or  a  fill-in-the-blanks  type  of  interface  may 
be  required.  For  the  expert  or  professional  user,  a  succinct  command  language  may  be  needed. 
Not  all  4GLs  provide  both  types  of  interfaces.  Those  that  do  may  not  necessarily  provide  all  of 
the  functionality  needed  for  the  application. 

These  application  requirements  are  used  to  develop  a  hst  of  desired  4GL  features,  step  five  of  the 
selection  process.  These  features  must  be  rated  by  importance  for  the  application  and  weights  must 
be  developed  for  this  rating.  Step  six  includes  identifying  mandatory  or  critical  features  that  the 
4GL  must  possess.  These  mandatory  or  critical  features  can  be  used  to  "narrow  the  field," 
eliminating  some  of  the  potential  4GL  products.  The  functional  benchmark  tasks  are  run  on  the 
few  products  that  remain  after  eliminating  those  that  do  not  possess  the  mandatory  or  critical 
features.  The  results  of  these  tasks  are  compared  with  the  weights  of  the  selection  feamres  to 
provide  the  final  selection  guidance. 

The  functional  benchmarks  are  divided  into  numerous  tasks.  Each  task  should  be  evaluated  on  the 
4GL's  ability  to  perform  the  task  and  the  ease  of  performing  the  task.  Determining  if  a  4GL  can 
perform  the  task  is  fairly  objective;  however,  judging  its  ease  of  use  is  more  subjective.  WhUe 
objectivity  in  developing  these  tests  was  a  significant  goal,  we  found  that  subjective  measures  were 
also  needed.    Judgements  such  as  a  preference  for  function  key-initiated  actions  as  opposed  to 
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spelled-out  command-initiated  actions,  or  color  monitor  screens  rather  than  black  and  white  screens, 
are  normally  realized  after  experience  has  been  gained  and  are  essentially  subjective  in  nature. 

Critical  requirements  and  the  required  level  of  functionality  provide  the  objective  measures  of  these 
tasks.  If  a  requirement  is  critical  (i.e.,  the  4GL  must  provide  the  functionaUty),  then  a  4GL  that 
does  not  provide  it  objectively  fails  the  test  for  that  capability.  If  the  capability  is  available,  but 
cannot  be  used  effectively  (e.g.,  procedures  may  contain  conditional  statement  constructs  such  as 
IF  statements,  but  contain  no  IF-THEN-ELSE  constructs),  then  you  may  decide  that  the  4GL  does 
not  meet  the  threshold  functionality  required,  and  it  may  faU  the  test. 

The  functional  benchmark  tasks  were  designed  to  mn  on  a  microcomputer  system  with  sufficient 
equipment  to  aUow  the  4GL  to  be  installed.  They  could,  however,  be  run  on  a  larger  computer 
if  desired.  We  have  endeavored  to  make  the  tasks  as  generic  as  possible  to  remove  portabihty 
problems  due  to  systems  software,  architecture  and  other  dependencies. 

While  a  production  system  based  on  the  selected  4GL  may  eventually  execute  on  a  minicomputer 
or  mainframe,  the  microcomputer  versions  generally  have  sufficient  capabilities  to  allow  testing  of 
functionality  and  provide  a  cost  effective  instrument  for  doing  so.  Once  a  4GL  has  been  selected 
for  an  application,  it  should  be  tested  on  the  hardware  that  will  be  used  before  it  is  put  into 
production. 

2.1  Benchmarking  Process 

There  are  six  steps  in  the  process  of  performing  these  benchmark  tasks.  They  are~ 

1.  Rate  the  desired  features  of  a  4GL  according  to  the  application's  requirements  and 
assign  appropriate  weights  to  these  features. 

2.  Select  the  functional  benchmark  tasks  which  are  appropriate  for  measuring  these 
desired  features. 

3.  Perform  the  appropriate  tasks. 

4.  Evaluate  the  4GL's  performance  on  each  task. 

5.  Compute  the  4GL's  weighted  score. 

6.  Analyze  the  results. 

2.1.1  Rate  4GL  Features  and  Assign  Weights 

The  importance  of  a  particular  requirement  is  determined  by  its  weight  in  relation  to  other 
requirements.  You  assign  these  weights  as  your  preference  before  tests  are  performed.  For 
example,  if  a  relational  database  capability  is  critical  for  your  applications,  the  weight  associated 
with  relational  capabilities  will  rank  very  high  on  your  preference  list.  On  the  other  hand,  graphics 
capabilities  may  play  no  role  in  your  requirements.  You  would,  therefore,  assign  littie  or  no 
importance  to  this  particular  aspect. 
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The  assignment  of  weights  to  requirements  may  be  performed  in  any  manner  deemed  appropriate 
to  an  organization's  philosophy  and  culture.  Some  organizations  provide  detailed  hsts  of  require- 
ments to  those  who  will  use  the  system  and  ask  for  ranked  preferences  for  each  requirement. 
Others  may  do  this  ranking  by  committee.  As  the  number  of  requirements  increases,  ranking  these 
requirements  gets  much  more  difficult. 

Another  suggested  method  of  assigning  weights  consists  of  using  a  generaUzed  requirements 
checklist  like  the  one  provided  in  this  report  (see  sec.  3).  Weights  of  importance  are  assigned 
using  any  numbering  system.  Often  numbers  from  one  to  five  are  used  with  the  number  five 
having  near-critical  importance  and  the  number  one  having  little  importance.  One  user  or  several 
users  could  assign  the  weights.  If  several  users  assign  weights,  an  individual  must  be  assigned  to 
develop  one  overall  or  average  weight  for  each  requirement.  Requirements  that  are  not  germane 
would  be  rated  zero. 


2.1.2  Select  Appropriate  Tasks 

You  should  identify  which  tasks  are  to  be  performed  depending  on  the  requirements  that  you  have 
specified.  The  tasks  that  relate  to  the  requirements  are  identified  by  section  number  attached  to 
each  requirement.  The  number  of  tasks  that  should  be  run  are  totally  dependent  on  the  identified 
requirements  of  the  application.  If  the  apphcation  does  not  require  specific  functions,  these  tasks 
should  not  be  executed. 


2.13  Perform  the  Appropriate  Tasks 

We  suggest  that  you  run  the  tasks  in  the  order  they  are  presented.  For  example,  the  4GL  must  be 
installed  before  files  or  tables  can  be  defined  using  the  4GL.  We  strongly  suggest  that  you  adjust 
the  tasks  to  meet  the  specific  requirements  of  your  application  and  use  files  required  by  the 
apphcation. 


2.1.4  Rate  the  Performance  of  Each  Task 

As  you  perform  each  task,  you  should  rate  the  results  according  to  how  much  of  the  task  was 
accomplished  and  how  well  the  4GL  performed  the  task.  There  are  two  numeric  ratings  given  to 
each  result.  The  first  rating  is  the  functional  rating,  or  how  much  of  the  task  was  accompUshed 
(e.g.,  0-none,  1-some,  2-most,  3-all).  The  second  describes  how  well  you  think  the  4GL  performed 
the  task  or  how  easy  it  was  for  you  to  perform  the  task  (e.g.,  0-not  done/impossible,  1 -poor- 
ly/hindrance, 2-adequately/helpful,  B-easily/very  helpful). 

Some  of  the  tasks  may  seem  difficult  to  rate  in  relation  to  functionality  and  ease  of 
use/performance.  Documentation,  for  example,  may  not  appear  to  have  a  measurable  performance, 
but  if  it  is  disorganized  or  incomplete,  you  may  find  that  it  hinders  progress  or  may  offer  no  help 
at  all.  In  this  case,  it  would  have  a  direct  impact  on  performing  a  specific  test. 

Since  the  tasks  are  grouped  by  requirement,  some  requirements  have  more  than  one  task.  In  some 
cases  this  is  because  the  requirement  cannot  be  tested  using  one  task  but  takes  multiple  steps  or 
multiple  tasks.  An  example  of  these  multistep  tasks  would  be  importing  ASCII  files.  First  a 
database  file  structure  must  be  described,  data  imported,  and  the  imported  data  compared  with  the 
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original  data.  Each  of  these  tasks  can  be  rated  but  they  are  grouped  under  the  requirement  of 
"Ability  to  import  ASCII  files."  Others  require  testing  various  aspects  of  the  requirement  to  obtain 
a  thorough  testing  of  several  variations.  For  example,  the  requirement  "Reporting  options  for  user 
specified  page  breaks"  has  four  tasks:  generate  a  report  with  a  page  break  after  each  total,  after 
each  subtotal,  after  40  lines,  and  after  a  sort  value  has  changed.  There  is  not  a  single  task  that 
adequately  tests  page  breaks. 

After  you  have  rated  these  multiple  tasks  under  a  requirement,  you  need  to  average  the  ratings  to 
get  one  average  value  for  functionality  and  one  for  ease  of  use  for  this  requirement.  Add  the 
functionality  ratings  for  each  task  and  divide  this  number  by  the  number  of  tasks  performed. 
Repeat  for  the  ease  of  use  ratings.  Enter  these  numbers  on  the  average  line. 

2.1^  Compute  the  Weighted  Score 

Once  the  selected  tasks  for  a  requirement  have  been  performed,  the  scores  are  totaled,  averaged, 
and  multiplied  by  the  weight  given  for  the  requirement.  These  weighted  numbers  are  then  added 
together  to  produce  a  total  weighted  value  for  the  4GL. 

2.1.6  Analyze  the  Results 

These  procedures  are  repeated,  with  the  same  weighted  requirements,  for  aU  4GL  products  under 
consideration.  The  total  weighted  value  for  each  4GL  tested  will  provide  an  indication  of  which 
4GL  will  most  closely  match  your  stated  requirements.  The  highest  scoring  product  should  fulfill 
the  majority  of  your  requirements. 

In  addition  to  looking  at  the  total  numbers  you  should  also  look  at  the  ratings  of  each  fimctional 
area  and  of  the  individual  requirements,  especially  for  the  highest  weighted  requirements.  This  is 
important  when  two  products  have  total  scores  that  are  very  close  to  each  other.  If  a  4GL  is  rated 
somewhat  higher  than  another,  but  you  do  not  feel  as  comfortable  about  the  higher  rated  one,  you 
may  be  able  to  determine  the  reasons  for  this  by  analyzing  the  scores  for  individual  functional  areas 
of  each  4GL.  An  analysis  of  variations  of  individual  area  scores  may  teU  you  how  much 
confidence  you  should  have  in  your  choice. 

2.2  Benchmarking  Organization 

The  requirements  and  tasks  are  divided  into  several  functional  areas  that  closely  parallel  the  feature 
areas  described  in  [GRAY86]: 

1.  operational  features; 

2.  data  management  feamres; 

3.  user  interfaces; 

4.  report  vmting  features; 

5.  language  features; 

6.  graphics  features; 

7.  security  features; 

8.  implementation  features;  and 

9.  application  development. 

This  organization  was  designed  to  facilitate  the  users'  normal  usage  of  a  4GL.  The  software  cannot 
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be  run  until  it  is  installed,  data  cannot  be  queried  until  the  database  exists,  etc.  This  logical 
grouping  of  functional  areas  can  be  described  as  follows. 

2.2,1  Operational  Features 

The  operational  features  are  those  functions  that  are  basic  to  the  operation  of  the  4GL.  In  general, 
operational  features  include  hardware  and  software  requirements  for  a  4GL,  portability  attributes, 
performance  characteristics,  communication  features  and  operating  system  interfaces.  Since  some 
of  these  features  are  critical  or  mandatory  features,  products  that  do  not  meet  these  requirements 
should  be  ehminated  before  getting  to  the  benchmarking  step  of  the  selection  process  (e.g.,  products 
that  wUl  not  run  on  the  hardware  required  would  not  get  to  the  benchmarking  step).  Thus,  tasks 
for  these  features  are  not  included  in  this  report. 

There  are  other  operational  features  that  cannot  be  evaluated  before  a  database  is  created  and 
experience  gained  with  the  4GL.  These  include  such  feamres  as  diagnostic  messages,  interfacing 
the  4GL  to  other  programming  languages,  and  quality  of  the  documentation.  These  features  are 
included  in  the  tasks  for  implementation  feamres. 

The  operational  features  included  in  this  section  of  requirements  and  tasks  are  those  that  relate  to 
getting  a  4GL  up  and  running:  installation  and  documentation  of  the  installation  procedures. 


2.2.2  Data  Management  Features 

The  general  data  management  functions  include  the  capabilities  necessary  to  manage  and  manipulate 
data;  to  define  data  structures,  store  data,  and  retrieve  data.  Because  a  4GL  has  other  capabOities 
to  manipulate  data  such  as  query  languages,  user  interfaces,  report  writing  features,  etc.,  this  section 
is  limited  to  data  definition  features,  file  creation,  importing,  and  exporting  data. 

The  ability  to  easOy  define  file  structures,  data  formats,  and  required  data  types  is  important  to 
the  capabihty  of  creating  new  databases.  The  ability  to  load  data  from  extemal  files  into  the 
database  can  be  critical  for  converting  ft"om  another  apphcation  or  in  accepting  data  created  by  other 
apphcations.  Since  many  4GLs  can  process  data  from  many  database  systems  without  conversion, 
the  ability  to  read,  write,  and  query  from  extemal  files  is  also  important  to  many  applications. 
These  functions  are  included  in  this  section. 


2.23  User  Interfaces 

There  are  many  different  kinds  of  user  interfaces  to  4GLs.  Menus  and  screens  are  some  of  the 
primary  means  that  a  user  has  to  interact  with  the  4GL  and  an  apphcation.  Some  language  features 
are  also  part  of  the  user  interface  but  they  are  included  in  another  section. 

The  abiUty  to  design  screens  and  menus  is  required  for  most  apphcation  development.  The  4GL 
may  need  to  provide  functions  such  as  field  character  validation,  field  masks,  required  fields,  filled 
fields,  computed  fields,  default  field  values,  screen/data  field  refresh,  interfield  checks,  and 
interrecord  checks  for  the  apphcation.  The  requirements  and  tasks  included  under  user  interfaces 
cover  the  fundamental  aspects  of  screen  and  menu  design. 
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2.2.4  Report  Writing  Features 


The  ability  to  display  or  print  both  detailed  and  summarized  data  in  a  straightforward  and 
meaningful  manner  is  one  of  the  most  important  features  of  a  4GL.  Generally,  a  4GL  provides  a 
default  report  format  for  simple  reports  so  that  the  user  can  issue  a  query  and  have  the  results 
printed  as  a  report  without  having  to  specify  elements  of  the  report  such  as  paging,  date,  page 
break,  column  position,  etc.  In  addition,  the  4GL  provides  capabilities  for  specifying  precisely  these 
elements,  if  the  user  desires.  The  most  complete  report  writer  will  provide  the  most  flexibility  for 
displaying  data. 


2.2.5  Language  Features 

There  are  many  language  functions  of  a  4GL:  nonprocedural  language  functions,  query  language 
functions,  programming  or  command  language  functions,  interfaces  with  other  programming 
languages,  and  the  ability  to  produce  compiled  or  optimized  code.  All  of  these  are  very  important 
to  the  operation  and  functionality  of  the  4GL.  Quite  often,  the  language  capabilities  determine 
whether  the  4GL  has  the  power  or  ease  of  use  that  is  desired  for  the  application.  If  the  user 
interface  is  difficult  to  use,  the  4GL  may  not  be  appropriate  for  novice  users.  If  it  lacks  some  of 
the  programming  capabilities  of  a  third  generation  language,  it  may  not  be  powerful  enough  for  the 
application.  The  richness  of  the  language  and  die  level  of  integration  between  all  aspects  of  the 
4GL  (report  writer,  screen  generator,  query  language)  strongly  influence  whether  or  not  the  software 
is  in  fact  a  fourth  generation  language. 

The  language  in  which  the  4GL  is  written  (the  implementation  code)  and  the  type  of  application 
code  it  produces  may  influence  the  portability  of  the  application  and  the  performance  of  the  4GL. 
These  features  of  the  language  are  difficult  to  generalize  into  tasks  but  should  be  considered  in  the 
overall  assessment  of  the  4GL. 

In  many  cases,  the  nonprocedural  language  of  the  4GL  is  implemented  in  screens  and  menus. 
Because  these  are  also  used  for  data  input,  display,  etc.,  the  tasks  for  testing  these  functions  are 
included  in  the  user  interface.  This  section  includes  requirements  and  tasks  for  query  language 
functions,  nonprocedural  language  functions,  procedural  language  functions,  and  integration  aspects 
of  the  language. 


2.2.6  Graphics  Features 

Some  4GLs  have  graphics  facilities  integrated  with  the  other  functions  of  the  4GL,  while  others 
provide  interfaces  to  separate  graphics  software.  Some  4GLs  simply  provide  a  third  generation 
language  interface  that  allows  existing  graphics  software,  such  as  FORTRAN  routines,  to  be 
executed.  The  requirements  of  the  application  have  to  be  matched  with  these  capabilities. 

There  is  a  wide  range  of  graphics  capabilities  that  are  available.  Some  4GLs  only  provide  simple 
business  graphics  capabilities,  bar  charts,  pie  charts,  and  line  graphs.  Others  may  provide  three- 
dimensional  graphs,  scatter  diagrams,  and  logarithmic  graphs.  The  most  important  feature  of  any 
of  these  capabilities  may  be  the  amount  of  integration  with  the  rest  of  the  4GL  functions.  For 
example,  creating  a  graph  whose  data  points  are  obtained  directly  from  the  database  and  whose 
labels  are  obtained  from  the  data  dictionary,  requires  much  less  effort  than  one  which  requires 
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typing  all  labels  or  re-entering  the  data. 

Novice  users  may  need  graphics  features  to  perform  scaling,  calculating  percentages  for  pie  charts, 
placing  labels,  and  formatting  bar  placement.  Users  may  want  th6  ability  to  view  the  chart  on  the 
screen  and  manipulate  the  results  before  any  printing  takes  place.  Some  users  may  want  high- 
quality  presentation  graphics,  high  resolution  capabilities,  color,  multiple  fonts,  and  interfaces  to 
plotters  or  photographic  output  devices. 

The  requirements  and  tasks  given  for  graphic  features  are  not  meant  to  cover  all  possible  graphic 
requirements.  If  an  application  has  very  complex  graphic  requirements,  additional  tasks  should  be 
added  for  the  benchmarking  process. 

2.2.7  Security  Features 

The  security  features  of  a  4GL  often  include  user  identification  and  password  protection  at  various 
levels  (user,  owner,  file  or  table,  record,  field,  entity  or  data  item,  procedure  [edit,  delete,  insert], 
etc.),  backup  protection  and  protection  of  data  during  crashes,  encryption  capabilities,  and  feamres 
for  audit  trails.  The  level  of  security  required  has  to  be  determined  for  the  application  and  the 
system  the  application  runs  on.  Often  the  system  security  or  pre-existing  software  provides  some 
of  the  needed  security  features.  If  adequate  security  software  already  exists,  then  the  software 
attributes  of  the  4GL  have  to  be  tested  for  secure  interfaces  to  ensure  that  they  do  not  provide  a 
"back  door"  into  the  system.  If  the  security  of  the  system  is  dependent  on  the  security  attributes 
of  the  4GL  then  these  features  of  the  4GLs  should  probably  be  more  thoroughly  tested  than  the 
tasks  that  we  provide.  The  tasks  given  for  this  area  cover  "typical"  kinds  of  security  functions  for 
4GLs  that  would  run  in  a  microcomputer  environment. 


2.2.8  Implementation  Features 

The  tasks  for  these  features  are  tasks  that  cannot  be  evaluated  until  the  user  has  gained  experience 
with  the  4GL.  As  mentioned  in  section  2.2.1,  many  of  these  tasks  are  related  to  operational 
features  such  as  interfacing  with  other  languages,  interfacing  with  the  operating  system,  quality  and 
usefulness  of  diagnostic  messages,  and  on-line  help  capabilities.  Others  are  related  to  overall  usage 
of  the  4GL,  such  as  judging  the  quality  of  the  documentation. 


2.2.9  Application  Development 

The  tasks  for  these  features  are  really  tasks  which  combine  or  incorporate  different  functions  of  the 
4GL  into  a  cohesive  application.  The  application  may  be  generated  from  the  screen  painter,  report 
generator,  and  language  feamres  which  have  been  tested  in  other  feature  areas.  This  grouping  of 
features,  however,  is  meant  to  test  the  ability  of  the  4GL  to  put  them  all  together  to  build  an 
application.  Language  integration,  language  functionality,  debugging  aids,  and  programming 
messages  are  examples  of  4GL  features  that  really  cannot  be  tested  except  by  building  an 
application  or  a  prototype  of  an  application. 
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3.  4GL  REQUIREMENTS  CHECKLIST 


The  4GL  requirements  checklist  must  be  used  for  the  first  procedure  in  the  benchmarking  process 
and  can  also  be  used  for  step  six  of  the  4GL  selection  process,  "develop  a  ranking  or  rating  system 
for  the  4GL  features  so  that  the  most  desired  features  can  be  differentiated  from  the  least  desired" 
[GRAY86].  If  the  checklist  is  being  used  for  step  six,  mandatory  features  should  be  identified  in 
this  step  plus  any  features  which  would  make  a  package  unacceptable.  First,  the  features  of  a 
4GL  that  are  critical  for  an  application  must  be  identified.  We  suggest  that  these  features  be 
identified  with  an  asterisk  (*),  meaning  that  the  application  would  not  be  acceptable  without  these 
features.  This  also  implies  that  a  product  would  be  unacceptable  for  use  if  it  did  not  possess 
these  features.  The  asterisks  represent  the  highest  rating  or  weighting  that  a  feature  can  receive  and 
should  be  used  to  "weed  out"  those  products  that  do  not  meet  the  application's  critical  requirements. 

If  the  checklist  is  being  used  for  the  benchmarking  process,  the  critical  features  have  already  been 
used  to  eliminate  those  products  that  do  not  meet  the  mandatory  requirements.  At  this  point  an 
asterisk  is  probably  not  the  most  useful  indicator.  The  critical  features  should  simply  be  assigned 
the  highest  rating. 

The  importance  of  the  other  4GL  features  relative  to  the  application  also  has  to  be  determined. 
Weights  for  these  other  features  should  be  assigned.  We  suggest  a  scale  of  one  to  five  with  five 
representing  the  highest  rating.  Features  with  a  weight  of  five  are  most  important  to  the 
application.  Features  with  a  weight  of  one  are  not  very  important.  These  might  be  nice  to  have 
but  are  not  the  most  important  features  that  are  required  of  the  application.  A  weight  of  zero 
would  imply  that  this  feature  is  not  required  by  the  application. 


ASSIGNMENT  OF  WEIGHTS  FOR  APPLICATION 

Weight 

1.  Operational  Features 

A.  Installation  management   

B.  Ease  of  installing  on  multiple  platforms   

2.  Data  Management 

A.  Basic  data  dictionary   

B.  Synonym  or  aliases   

C.  Headings,  titles,  or  labels   

D.  Comments  or  remarks   

E.  Date  formats   

F.  Time  formats   

G.  Money  formats   

H.  Variable-length  fields   

I.  Capable  of  handling  #  chars  per  field   

J.  Capable  of  handling  #  fields  per  record   

K.  Keyed  or  indexed  files   

L.  Data  structure  management   

M.  Ability  to  read,  query,  or  report  from  ASCII  files   

N.  Ability  to  import  ASCII  files   

O.  Ability  to  export  ASCII  files   
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p.  Ability  to  read,  query,  or  report  from  

formatted  files 

Q.  Ability  to  import  formatted  files 

R.  Ability  to  export  formatted  files 

S.  AbUity  to  modify  files  while  loading 
T.  File  check  while  loading 
U.  File  modification 

3i  User  Interface 

A.  Capability  to  create  simple  menus 

B.  Capability  to  create  complex  menus 

C.  Default  data  entry  screens  provided 

D.  Capability  to  create  straight  text  screens 

E.  Capability  to  create  display  screens 

F.  Data  checking  on  data  entry 

G.  Reverse  video  control 

H.  Blinking  data  entry  fields 

I.  Calculated  fields  display 

J.  Display  of  HELP  information  on  screen 

K.  CapabOity  to  generate  more  complex  screens 

L.  Ability  to  print  screen  layout  document 

4^  Report  Writing  Features 

A.  Default  report  formats  provided  for  paging,  dating, 
and  headings 

B.  Default  report  formats  provided  for  summary  functions 

C.  Default  report  selections  provided  for  ascending  sorts 

D.  Default  report  selections  provided  for  descending  sorts 
Complex  reporting  options  for: 


E. 

Row  totals  and  subtotals 

F. 

Percent  formats 

G. 

Zero  suppression 

H. 

Floating  dollar  signs 

1. 

Comma  option  for  numeric  fields 

J. 

User  specified  spacing 

K. 

User  specified  page  breaks 

L. 

Table  lookups 

M. 

Headings 

N. 

Footers 

0. 

Font  selection 

P. 

Odd  size  printouts 

Q. 

Forms  printing 

R. 

Suppressed  fields 

S. 

Calculated  fields 

T. 

Specific  field  placement 

U. 

Output  device  independence  (i.e.,  user  could  select 

screen,  printer,  or  file) 

V. 

Output  device  adjustments  (i.e.,  user  selects  screen. 

printer,  or  file  and  4GL  adjusts  for  page  size, 

screen  size,  etc.) 
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5i  Language  Features 

A.  SQL  implementation 

B.  Simple  queries  on  one  file 

C.  AND  OR  logic  -  one  file 

D.  Relational  join 

E.  AND  OR  logic  -  two  files 

F.  Complex  Boolean  logic 

G.  Traps  command  or  statement  errors 

H.  Facilitates  correction  of  errors 

I.  Compiles  code 
J.  Interprets  code 
K.  Condition  logic 
L,  Looping  logic 

M.  Intrinsic  date/time  functions  (DATE,  DAY, 

MONTH,  etc.) 
N.  Intrinsic  numeric  functions  (MINIMUM,  MAXIMUM, 

AVERAGE,  etc.) 

0.  Statistical  ftmctions: 

Standard  deviation 
Variance 

Regressional  analysis 
Correlation 
Discriminant  analysis 
Factor  analysis 
Descriptive  statistics 
P.  Ability  to  process  missing  data 

6i  Graphics  Features 

A.  Ability  to  create  bar  graphs 

B.  Ability  to  create  pie  charts 

C.  Ability  to  create  exploded  pie  charts 

D.  Ability  to  create  scatter  diagrams 

E.  Ability  to  create  line  graphs 

F.  Ability  to  create  connected  point  plots 

G.  Ability  to  incorporate  trend  lines  in  scatter  diagrams 

H.  Ability  to  curve  fit  or  smooth 

1.  Availability  of  automatic  scaling 
J.  Availability  of  logarithmic  scaling 

K.  Ability  to  generate  field  labels  and  titles  from  the  data 

dictionary 
L.  Ability  to  utilize  multiple  fonts 
M.  Ability  to  calculate  values  to  incorporate  in  the  graph 
N.  Grid  option 
O.  Color  option 
P.  Hatch  pattem  options 
Q.  Ability  to  process  missing  values 

7.  Security  Features 

A.  Ability  to  specify  user  access 
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B.  Ability  to  specify  file  security 

C.  Ability  to  limit  command  or  procedure  usage 

D.  Ability  to  specify  record  access 

E.  Ability  to  specify  field  access 

F.  Ability  to  encrypt/decrypt  files 

G.  Ability  to  encrypt/decrypt  data  dictionary 

H.  Default  logon  procedure 

I.  User  profile 

J.  Ability  to  define  system  administrator 

K.  Feature  restrictions 

L.  Ability  to  create  a  user  logon  audit  file 

8i  Implementation  Features 

A.  Interface  to  other  software  packages 

B.  Interface  to  other  languages 

C.  Interface  to  operating  system 

D.  POSIX  compatibility 

E.  Online  HELP 

F.  Documentation 

G.  Interface  to  external  editor/word  processor 

H.  Error  messages  and  wamings 

9i  Application  Development 

A.  Program  editor 

B.  Error  messages 

C.  Branching  logic 

D.  Looping  logic 

E.  Language  integration 

F.  Ability  to  call  subroutines 

G.  Ability  to  imbed  queries  in  procedures 

H.  Ability  to  share  temporary  data 

I.  Ability  to  create  global  variables 
J.  Ability  to  create  local  variables 

K.  Ability  to  define  calculations  using  field  values 


4.  BENCHMARKING  TASKS 

The  functional  benchmarks  are  divided  into  numerous  tasks.  The  tasks  are  organized  by  the 
groupings  described  previously.  The  tasks  are  listed  by  the  appUcation  requirement  number, 
following  the  same  order  as  the  requirements  checklist. 

Each  task  should  be  evaluated  on  the  4GL's  functional  ability  to  perform  the  task  and  the  ease  of 
performing  the  task.  We  suggest  a  rating  scale  of  zero  to  three  for  this  process.  For  functionality, 
a  4GL  that  could  perform  all  of  the  functionahty  that  was  required  to  accomplish  the  task  would 
receive  a  three.  If  most  of  the  task  could  be  accomplished,  a  two;  some,  a  one;  and  if  none  of 
the  task  could  be  completed,  the  4GL  would  score  a  zero.  For  ease  of  use/performance,  if  the  task 
was  completed  easily  and  the  4GL  facilitated  the  process,  the  4GL  would  score  a  three.  If  the  task 
was  extremely  difficult  to  accomplish  the  4GL  might  score  a  one.  If  it  was  impossible  to  complete. 
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it  would  score  a  zero.  A  summary  of  this  suggested  rating  scale  is  provided  below  and  at  the 
beginning  of  each  section. 


RATING 
SCALE 


Functionality 

0-  none 

1-  some 

2-  most 

3-  alI 


Performance/Ease  of  use 

0-  not  done/impossible 

1-  poorly /hi  ndrance 

2-  adequately/helpful 

3-  easily/very  helpful 


4.1  Operational  Features 

Any  product  must  be  installed  before  you  can  test  its  capabilities.  Some  products  are  much  more 
difficult  to  install  than  others.  Because  all  products  are  different,  we  cannot  specify  commands  for 
a  specific  type  of  instaUation.  You  must  follow  the  directions  supphed  with  the  product. 

Your  goal  during  this  process  is  to  evaluate  the  difficulty  of  the  installation,  how  well  the 
documentation  matches  what  actually  occurs,  if  the  documentation  provides  enough  detail,  if  the 
storage  requirements  are  as  advertised,  and  how  difficult  it  is  to  recover  ft-om  any  errors  that  may 
occur  during  installation. 

Some  applications  may  require  individual  installation  at  numerous  sites  with  different  personnel 
performing  installation.  In  this  case,  even  more  attention  should  be  paid  to  evaluation  results  than 
what  we  suggest.  For  applications  that  wiU  require  only  one  installation  and  occasional  upgrades, 
the  additional  tasks  for  multiple  installation  sites  should  not  be  required. 


RATING 
SCALE 


Functionality 

0-  none 

1-  some 

2-  most 

3-  all 


lA.  Installation  management 
1.  Install  the  4GL. 


Performance/Ease  of  use 

0-  not  done/impossible 

1-  poorly/hindrance 

2-  adequately/helpful 

3-  easily/very  helpful 


FUNC.  EASE 


2.  Rate  the  documentation  (user's  guide,  reference  manual,  etc.) 
for  installation  instructions. 


Rate  the  ability  to  customize  the  installation,  (i.e.,  does  the  user 
have  the  option  on  whether  or  not  to  include  certain  features, 
such  as  the  demonstration/tutorial  or  the  graphics  feature?) 


Total 


Average  (Total/3)    (lA) 

IB.  Ease  of  installing  on  multiple  platforms 

Install  the  4GL  on  a  second  system  with  a  different 

operating  system  or  different  hardware.     (IB) 


15 


4.2  Data  Management 


A  4GL  contains  many  capabilities  for  managing  and  manipulating  data.  The  first  of  these  is  the 
definition  of  data  structures  to  be  stored  in  the  database.  This  set  of  tasks  wiU  help  determine  if 
the  4GL  contains  appropriate  functionahty  to  define  and  manage  the  data  structures  needed  for  the 
planned  applications. 

The  second  major  capability  under  the  data  management  heading  is  that  of  data  loading.  A  separate 
section  is  devoted  to  testing  this  aspect.  The  ability  to  load  data  from  external  files  into  the 
database  wiU  assist  in  converting  ft"om  another  application  or  in  accepting  data  created  by  other 
apphcations.  We  suggest  that  data  from  an  existing  appUcation  be  used  to  perform  the  data  loading 
tests.  Since  the  personnel  performing  the  evaluation  should  be  familiar  with  that  data,  errors  in  test 
performance  will  be  more  readily  visible  than  if  "made-up"  data  were  used. 

If  no  specific  data  comes  to  mind  for  performing  these  tests,  we  suggest  generating  at  least  two 
files  containing  data  appropriate  for  the  planned  applications.  We  suggest  using  a  large  file  with 
two  to  three  thousand  records,  and  a  smaller  file,  with  each  record  containing  at  least  several 
alphanumeric  data  elements  and  several  types  of  numeric  data  elements. 

At  first  glance,  it  may  appear  that  some  of  the  tests  can  be  combined.  However,  evaluators  may 
run  into  problems  with  different  4GLs  due  to  limitations  of  each  particular  4GL  in  performing 
combined  tasks.  Errors  encountered  in  evaluation  may  be  due  to  combinations  of  effects.  In  such 
cases,  it  may  be  difficult,  if  not  impossible,  to  ascertain  what  the  real  problem  is  and  how  to 
overcome  it.  For  this  reason,  the  tests  are  specified  individually  to  control  as  much  extemal 
influence  as  possible. 


Functionality  Performance/Ease  of  use 

RATING        0-none  0-not  done/impossible 

SCALE  1-some  1-poorly/hindrance 

2-  most  2-adequately/helpful 

3-  aII  3-easiIy/very  helpful 


2A.  Basic  data  dictionary  FUNC.  EASE 

1.  Using  whatever  method  is  suggested  by  the  vendor  (e.g.,  screens,  forms, 

prompts,  etc.),  define  a  sequential  file  (FILE  A).     

2.  Define  a  file  with  at  least  one  key  or  index  made  up  of  at  least  one 

data  element  (FILE  B).     

3.  Define  a  numeric  data  element  containing  a  decimal  point.     

4.  Define  an  integer  data  element.     

5.  Define  a  data  element  containing  only  alphabetic  characters.     

6.  Define  a  data  element  containing  both  alphabetic  and  numeric  characters.     
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7.  List  all  information  pertaining  to  FILE  A  and  FILE  B  including  the  file 
name,  file  organization,  data  element  names,  data  element  types,  and 
sizes  on  the  terminal  screen. 

8.  List  all  information  pertaining  to  FILE  A  and  FILE  B  including  the  file 
name,  file  organization,  data  element  names,  data  element  types,  and 
sizes  on  a  printer. 

Total 

Average  (Total/8) 

2B.  Synonym  or  aliases 

Define  a  synonym  or  alias  for  a  data  element  in  FILE  A. 

2C.  Headings,  titles,  or  labels 

Define  a  heading,  titie,  or  label  for  a  data  element  in  FILE  A. 

2D.  Comments  or  remarks 

Define  a  comments  or  remarks  section  for  the  file  or  a  data  element 
in  RLE  A. 

2E.  Date  formats 

Define  a  data  element  in  FILE  A  with  a  date  format. 

2F.  Time  formats 

Define  a  data  element  in  FILE  A  with  a  time  format. 

2G.  Money  formats 

Define  a  data  element  in  FILE  A  with  a  money  format. 

2H.  Variable-length  fields 

Define  a  data  element  in  FILE  A  with  a  variable  length. 

21.  Capable  of  handling  #  chars  per  field 

Define  a  numeric  element  with  the  size  of  the  largest  number  that  will 
most  likely  be  used  in  an  application  or  a  character  element  with 
the  largest  number  of  characters  that  will  be  used. 

2J.  Capable  of  handling  #  fields  per  record 

Define  a  sufficient  number  of  data  elements  in  FILE  A  and  FILE  B  to 
determine  if  the  4GL  can  accommodate  the  largest  number  of  fields 
required  per  record. 

2K.  Keyed  or  indexed  files 

1.  Define  a  sequential  file  (FILE  C). 

2.  Define  a  file  with  at  least  one  key  or  index  made  up  of  at  least  one 
data  element  (FILE  D). 
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3.  Define  a  key  in  FILE  D  that  is  the  same  as  the  key  in  FILE  C. 

4.  Define  a  key  in  FILE  D  that  is  different  from  all  previous  keys. 

5.  Define  a  key  in  FILE  C  that  is  made  up  of  at  least  two  concatenated 
data  elements. 

Total 

Average  (Total/5) 

2L.  Data  structure  management 

1.  Delete  a  non-key  data  element  from  FILE  D. 

2.  Delete  a  key  data  element  from  FILE  D. 

3.  Delete  a  data  element  from  FILE  C. 

4.  Delete  FILE  C. 

5.  Copy  FILE  A  description  to  FILE  E. 

6.  Copy  FILE  B  description  to  FILE  F. 

7.  Delete  FILE  E  and  FILE  F. 

8.  Edit  a  data  description  by  changing  the  spelling  of  a  field/data  element 
name. 

9.  Change  the  format  of  a  field/data  element. 

Total 

Average  (TotaV9) 

2M.  Ability  to  read,  report,  or  query  from  ASCII  files 

1.  Define  a  file  structure  for  an  external  ASCII  file  (FILE  1)  to  be  loaded. 

2.  Read  the  external  file  and  create  a  report  or  query  from  the  extemal 
file's  data  without  creating  a  4GL  formatted  fUe.  (If  you  could  not 
create  a  report  or  query  from  the  extemal  file,  list  the  first  10  and 
last  10  records  of  the  extemal  file  using  any  means  appropriate.) 

Total 

Average  (Total/2) 

2N.  Ability  to  import  ASCII  files 
1.  Define  a  file  structure  for  a  4GL  database  that  will  accommodate  FILE  1. 
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2.  Load  the  data  from  the  external  file  into  the  4GL  database  structure 
creating  the  4GL  internal  file,  FILE  2. 

3.  To  ensure  that  the  data  is  loaded  correctly,  repeat  the  same  report  or 
query  that  you  created  in  task  2M  and  compare  the  data.  (If  you  could 
not  create  a  report  or  query  from  an  external  file,  list  the  first  10  and 
last  10  records  of  the  database  file  (FILE  2)  using  the  4GL.  Compare 
the  data  in  this  report  with  the  data  in  the  external  file.) 

Total 

Average  (Total/3) 

20.  Ability  to  export  ASCII  files 

Save  the  results  of  the  previous  query  to  an  external  ASCII  file.  Type 
or  print  the  external  file  to  compare  results. 


2P.  Ability  to  read,  report,  or  query  from  formatted  files 

Repeat  tasks  2M  for  all  file  formats  required. 


2Q.  Ability  to  import  formatted  files 

Repeat  tasks  2N  for  aU  file  formats  required. 

2R.  Ability  to  export  formatted  files 

Repeat  tasks  20  for  all  file  formats  required. 

2S.  Ability  to  modify  files  while  loading 

1.  Define  an  intemal  file  (FILE  3)  stmcture  similar  to  the  external  file 
structure  of  FILE  1,  and  augment  it  with  at  least  one  data  element  that 
is  not  in  the  extemal  file  structure. 

2.  Load  data  from  the  extemal  fiile  (FILE  1)  into  the  augmented  4GL 
database  file  (HLE  3). 

3.  Define  an  intemal  file  (FILE  4)  structure  similar  to  the  extemal  file 
structure  of  FILE  1,  but  rearrange  the  order  of  the  data  elements. 

4.  Load  the  data  from  the  extemal  file  (FILE  1)  into  the  rearranged  4GL 
file  (FILE  4). 

Total 

Average  (Total/4) 
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2T.  File  check  while  loading 

Load  descending  sorted  data  from  the  external  file  into  an  ascending 

sorted  internal  file  without  first  deleting  the  data  from  the  internal  file. 

(Rate  the  functionality  3,  if  the  duplicates  are  rejected.)     (2T) 

2U.  File  modification 

1.  Display  basic  attributes  of  a  file,  including  the  file  size,  number  of 
records,  time  the  file  was  created,  time  the  file  was  last  modified  or 
accessed.     


2.  Display  file  usage  information  including  number  of  records  added, 
deleted,  or  modified. 

3.  Add  a  record  to  the  file.  Display  the  record. 

4.  Modify  a  record  in  the  file.  Display  the  record. 

5.  Delete  a  different  record  in  the  file. 


6.  List  the  attributes  of  the  file  including  the  file  size,  number  of  records, 
time  the  file  was  created,  time  the  file  was  last  modified  or  accessed. 
Make  certain  that  the  changes  were  reflected  in  the  hsting. 

7.  Display  the  file  usage  information  including  number  of  records  added, 
deleted,  or  modified  to  make  certain  that  the  changes  were  reflected 
in  the  Usting. 

Total 


Average  (Total/7)    (2U) 

4.3  User  Interface 

Screens  and  menus  are  the  primary  means  that  a  user  has  of  interacting  with  an  appUcation.  For 
this  reason,  a  4GL  must  have  a  diversified  and  robust  set  of  screen  formatting  capabilities.  These 
capabilities  are  generally  grouped  in  two  categories:  screen  formatting  commands  and  data 
manipulation  commands.  Screen  painters  are  often  used  to  implement  these  capabilities  within  a 
4GL.  Text,  symbols,  and  data  input/output  areas  are  defined  on  a  terminal  screen  through 
placement  of  a  cursor  and  the  use  of  predefined  commands  or  control  keys. 


Functionality  Performance/Ease  of  use 
RATING         0-none  0-not  done/impossible 

SCALE  1-some  1-poorly/hindrance 

2-  most  2-adequately/helpful 

3-  aIl  3-easily/very  helpful 


3A.  Capability  to  create  simple  menus  FUNC.  EASE 

1.  Define  a  default  menu  with  at  least  two  selections.     
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2.  Define  a  second  default  menu  to  be  executed  as  one  of  the  choices 
of  the  first  menu. 


Total 

Average  (Total/2) 

3B.  Capability  to  create  complex  menus 

1.  Define  a  vertical  menu  containing  at  least  two  selections. 

2.  Define  a  horizontal  menu  containing  at  least  two  selections. 

3.  Define  a  menu  that  uses  function  keys  or  control  keys  as  the  means 
of  selecting. 

4.  Define  a  menu  that  uses  cursor  positioning  as  the  means  of  selecting. 

5.  Define  a  menu  that  uses  character  strings  or  numbers  as  the  means 
of  selecting. 

6.  Define  a  pull-down  menu  with  at  least  two  selections. 

7.  Define  a  menu  with  text,  menu  background,  and  menu  border  in 
different  colors. 

8.  Define  a  menu  with  more  entries  than  can  be  displayed  on  the  screen 
at  one  time. 

9.  Define  a  menu  that  scrolls  according  to  the  location  of  the  selection 
cursor  (i.e.,  on  a  vertical  menu,  some -of  the  selections  may  not  be 
visible  through  scrolling  of  the  selections  off  the  top  or  bottom  of  the 
menu.  A  horizontal  menu  may  scroll  to  the  left  and  right). 

10.  Define  a  menu  that  allows  speed  searching  of  the  selections  (i.e.,  the 
selection  cursor  advances  to  the  next  selection  starting  with  the 
character  entered  by  the  user). 

Total 

Average  (Total/10) 

3C.  Default  data  entry  screens  provided 

Access  a  default  data  display  screen  containing  at  least  two  data 
elements  and  the  names  associated  with  the  data  elements. 

3D.  Capability  to  create  straight  text  screens 

Define  an  information  screen  containing  text  only  (no  records  or 
variables). 
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3E.  Capability  to  create  display  screens 

1.  Define  a  screen  and  associated  process  to  display  a  record  in  a  file 
when  the  screen  is  executed. 

3F.  Data  checking  on  data  entry 

1.  Access  a  default  data  entry  screen  containing  at  least  two  data  element 
input/display  areas  and  the  names  associated  with  the  data  elements. 
(These  data  elements  may  be  data  elements  defined  in  a  previous  test 
for  file  definition  or  data  loading.) 

2.  Enter  an  alphanumeric  string  in  a  numeric  data  entry  area  to  ascertain 
if  the  data  is  accepted  or  not. 

3.  Enter  a  numeric  string  in  an  alphanumeric  data  entry  area  to  ascertain 
if  the  data  is  accepted  or  not. 

4.  Enter  a  non-date  string  in  a  date  entry  area  to  ascertain  if  the  data  is 
accepted  or  not. 

5.  Using  a  data  element  for  which  an  entry  is  required,  try  to  bypass  the 
field  without  entering  a  value. 

6.  Using  a  data  element  for  which  a  default  entry  is  defined,  bypass  the 
entry  field  without  entering  a  value  to  determine  if  the  default  value 
has  been  automatically  displayed. 

Total 

Average  (Total/6) 

3G.  Reverse  video  control 

Define  a  reverse  video  data  entry  area. 

3H.  Blinking  data  entry  fields 

Define  a  blinking  data  entry  area. 

31.  Calculated  fields  display 

Define  a  data  display  area  with  the  contents  computed  from  other  data 
entry  areas  (i.e.,  such  as  a  total,  balance,  or  extended  amount). 

3J.  Display  of  HELP  information  on  screen 

1.  Define  a  data  display  window  that  contains  help  information  for  a 
particular  data  entry  area. 

2.  Define  a  help  display  area  on  screen. 

Total 

Average  (Total/2) 
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3K.  Capability  to  generate  more  complex  screens 

1.  Define  a  screen  containing  at  least  one  display-only  data  element,  one 
data  entry  element,  one  hidden  data  element,  and  one  update-only  data 
element  for  each  of  these  data  types:  integer,  floating  point, 
alphanumeric,  money,  and  date/time. 

2.  Define  a  display  area  on  the  screen  that  remains  hidden  until  needed,  as 
opposed  to  an  area  that  is  visible  on  screen  at  all  times. 

3.  Define  an  array  data  entry  area  (i.e.,  multiple  columns  or  rows  with 
repeating  data  elements,  such  as  may  be  found  on  a  multiple-line-item 
purchase  order  or  parts  list). 

Total 


Average  (Total/3)  (3K) 

3L.  Ability  to  print  screen  layout  document 

Document  tiie  screen  format  by  printing  a  copy  of  the  screen 

commands  or  screen  layout  on  a  printer.     (3L) 


4.4  Report  Writing  Features 

Generally,  a  4GL  provides  a  default  report  format  for  simple  reports  so  that  the  user  can  issue  a 
query  and  have  the  results  printed  as  a  report  without  having  to  specify  elements  of  the  report  such 
as  paging,  date,  page  break,  column  position,  etc.  These  attributes  are  tested  first.  In  addition,  the 
4GL  provides  capabilities  for  specifying  detailed  report  elements,  if  the  user  desires.  Tasks  are 
included  for  testing  a  number  of  these  capabilities.  Only  those  tasks  that  are  required  for  the 
application  need  to  be  executed. 

Functionality  Performance/Ease  of  use 

RATING        0-none  0-not  done/impossible 

SCALE  1-some  1-poorly/hindrance 

2-  most  2-adequately/helpful 

3-  all  3-easily/very  helpful 

4A.  Default  report  formats  provided  for  paging,  dating,  and  headings 

Using  as  many  default  instructions  as  possible,  generate  a  report 

including  at  least  two  data  elements.  Specify  only  the  data  elements 

to  be  printed.  Let  the  4GL  determine  element  placement,  paging,  and 

headings.     (4A) 

4B.  Default  report  formats  provided  for  summary  functions 

1.  Generate  a  default  report  including  a  character  element  and  at  least  one 
numeric  data  element  and  compute  subtotals  and  totals  on  this 

numeric  data  element.     


2.  Generate  a  default  report  that  counts  the  number  of  records  per  group 
(subsort)  and  per  report. 
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3.  Generate  a  default  report  that  computes  the  average  value  of  a  numeric 
column  by  group  and  by  report. 

Total 

Average  (Total/3) 

4C.  Default  report  selections  provided  for  ascending  sorts 

Generate  a  default  report  including  at  least  one  character  data  element 
and  specify  an  alphabetic  or  ascending  sort  on  this  data  element. 

4D.  Default  report  selections  provided  for  descending  sorts 

Generate  a  default  report  including  at  least  one  numeric  data  element 
and  specify  a  descending  sort  on  this  data  element. 

4E.  Complex  reporting  options  for  row  totals  and  subtotals 

Create  a  report  with  multiple  columns  of  numeric  data  elements. 
Generate  and  write  row  totals  and  subtotals. 

4F.  Reporting  options  for  percent  formats 

Compute  a  percentage  of  two  columns  by  using  a  built-in  percent 
function  and  total  the  results. 

4G.  Reporting  options  for  zero  suppression 

Display  a  number  that  is  less  than  the  maximum  number  of  digits  that 
can  be  displayed  and  suppress  the  leading  zeros. 

4H.  Reporting  options  for  floating  dollar  signs 

Display  a  number  in  a  numeric  data  element  with  currency  format. 
Multiply  the  number  by  10  and  by  1000.  Print  the  results. 

41.  Comma  option  for  numeric  fields 

Create  a  report  with  at  least  one  numeric  field.  Display  the  field  with 
commas  separating  every  three  digits.  Display  the  field  without  commas. 

4J.  Reporting  options  for  user  specified  spacing 

Generate  a  report  that  is  single  spaced  with  a  blank  line  after  each 
subtotal  and  containing  two  blank  lines  after  each  total. 

4K.  Reporting  options  for  user  specified  page  breaks 

1.  Generate  a  report  with  a  page  break  after  each  total. 

2.  Generate  a  report  with  a  page  break  after  each  subtotal. 

3.  Generate  a  report  with  a  page  break  after  40  lines. 


24 


4.   Generate  a  report  with  a  page  break  after  the  value  of  a  sort  element 
has  changed. 

Total 

Average  (Total/4) 

4L.  Reporting  options  for  table  lookups 

Create  a  report  with  a  field  that  is  referenced  or  cross-referenced  ft^om 
another  file.  (An  example  would  be  to  list  a  column  with  the  state 
name  when  the  main  file  only  has  state  codes.) 

4M.  Ability  to  generate  report  headings 

1.  Create  a  report  with  a  header  that  appears  only  at  the  beginning  of  the 
report. 

2.  Create  a  report  with  a  header  that  is  displayed  at  the  top  of  each  page 
of  the  report. 

3.  Create  a  report  with  a  header  that  is  printed  each  time  the  grouping 
value  changes,  (i.e.,  for  a  number  of  records  that  are  grouped  by  a 
common  field  value,  a  header  that  is  displayed  before  each  group). 

Total 

Average  (TotaV3) 

4N.  Ability  to  generate  report  footers 

1.  Create  a  report  with  a  footer  that  is  displayed  only  at  the  end  of  the 
report. 

2.  Create  a  report  with  a  footer  that  is  printed  at  the  bottom  of  each  page 
of  the  report. 

3.  Create  a  report  with  a  footer  that  is  printed  after  each  group  (i.e.,  for 

a  number  of  records  grouped  by  a  common  field  value,  a  footer  that  is 
displayed  after  the  last  record  in  each  group). 

Total 

Average  (Total/3) 

40.  Reporting  options  for  font  selection 

Create  a  report  with  multiple  fonts  (e.g.,  large  font  for  headings, 
smaller  font  for  detail  lines). 

4P.  Reporting  options  for  odd  size  printouts 

Create  a  report  which  exceeds  the  normal  page  size  (e.g.,  220  characters 
across  the  page  and  90  lines  per  page). 
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4Q.  Reporting  options  for  forms  printing 

Create  a  form  that  incorporates  line  drawings  with  data  from  a  database. 

Print  the  form  with  the  data.     (4Q) 


4R.  Reporting  options  for  suppressed  Helds 

1.  Create  a  report  that  is  sorted  on  one  field.  Sort  the  report  by  that  field 
and  suppress  printing  the  field  when  repeated  values  occur, 

2.  Using  the  same  report,  alter  the  setting  so  the  sort  field  is  printed  on 
every  occurrence. 

Total 


Average  (Total/2)    (4R) 

4S.  Reporting  options  for  calculated  fields 

Create  a  report  with  a  numeric  field  value.  Define  a  field  in  the  report 

that  is  the  result  of  some  calculation  involving  the  first  field.     (4S) 

4T.  Specific  field  placement 

Define  a  report  with  specified  field  placement  (i.e.,  the  user  specifies  x-y 

coordinate  or  column-row  values  for  field  placement  rather  than  using 

default  values).     (4T) 

4U.  Output  device  independence 

Using  any  of  the  reports,  print  the  report  to  the  printer.  Display  the 

same  report,  without  having  to  respecify  the  contents,  on  the  screen.     (4U) 

4V.  Output  device  adjustments 

Using  any  report  of  at  least  100  lines,  print  the  report  to  the  printer 

and  display  it  on  the  screen.  Did  the  4GL  adjust  the  page  breaks,  wrap 

arounds,  headings,  etc?     (4V) 


4.5  Language  Features 

The  goal  in  this  section  is  to  test  the  language  capabilities  of  the  4GL  that  are  appropriate  for  your 
requirements.  Keep  in  mind  that  the  requirements  may  change  over  time.  For  example,  a  friendly 
novice  level  interface  is  nice  in  the  beginning  until  users  know  the  4GL.  After  the  4GL  has  been 
used  for  a  while,  the  novice  level  interface  may  become  burdensome  and  cumbersome.  It  becomes 
aggravating  to  have  to  go  through  ten  menus  to  select  choices  when  you  need  only  to  enter  three 
commands.  Thus,  you  need  to  analyze  the  requirements  to  insure  that  the  4GL  has  the  flexibility 
you  will  need  not  only  from  an  initial  perspective  but  also  fi*om  a  long  range  perspective. 

This  section  mainly  tests  the  query  language  or  command  language  functions  of  a  4GL.  Some 
4GLs  have  a  separate  procedural  language  that  is  for  developing  programs,  or  have  added 
syntax/commands  for  programming  logic.  We  have  tried  to  separate  the  two  where  possible. 
Technically  the  functions  may  be  similar  but  the  ease  of  use  may  be  quite  different.  For  example, 
some  4GLs  have  a  condition  statement  that  can  be  added,  often  by  menu  selection,  to  a  query  or 
report,  e.g.,  IF  code  =  123,  WHERE  salary  <50000,  FOR  name  =  "Smith",  etc.,  which  is  different 
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from  programming  IF.. THEN.. ELSE  logic  to  specify  the  execution  of  different  sections  of  a 
program.  The  ease  of  use  rating  might  be  very  different.  The  programming  language  functions 
are  tested  in  the  application  development  section. 

Functionality  Performance/Ease  of  use 

RATING        0-none  0-not  done/impossible 

SCALE  l-some  1-poorIy/hindrance 

2-  most  2-adequateIy/helpful 

3-  all  3-easiIy/very  helpful 


5 A.  SQL  implementation 

1.  Complete  tasks  5B  through  5F  using  SQL  commands.  (See  [FIPS127] 
"Database  Language  SQL.")  Rate  this  task  for  overall  implementation 
and  integration  with  the  rest  of  the  4GL.  If  you  want  to  test  other 
query  capabilities  of  the  4GL  (e.g.,  non-SQL  interface)  repeat  these 

tasks  and  record  a  second  set  of  numbers.     (5A) 

5B.  Simple  queries  on  one  file 

1.  Query  a  file  to  find  all  records  containing  a  specific  non-null  string.     

2.  Query  a  file  to  find  all  records  not  containing  a  specific  non-null  string 

(the  complement  of  the  previous  test).     

3.  Query  a  file  to  find  all  records  within  a  specified  range.     

Total     


Average  (Total/3)    (5B) 

5C.  AND  OR  logic  -  one  file 

1.  Query  a  file  to  find  all  records  containing  one  data  element  value  AND 

another  element  value.     


2.   Query  a  file  to  find  all  records  containing  one  data  element  value  OR 
another  element  value. 


Total     (5C) 

Average  (Total/2)    (5C) 

5D.  Relational  join 

Relationally  join  two  files  without  creating  a  third  physically  merged  file.    (5D) 

5E.  AND  OR  logic  -  two  files 

1.   Query  the  joined  files  to  find  all  records  containing  one  data  element 
value  from  one  file  AND  one  data  element  value  from  the  other  file 
(i.e.,  A  AND  B).     
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2.  Using  the  same  joined  files,  query  the  files  to  find  all  records  containing 
one  data  element  value  fi-om  one  file  OR  another  data  element  value 
ft-om  the  other  file  (i.e.,  A  OR  B). 

Total 

Average  (Total/2) 

5F.  Complex  Boolean  logic 

1.  Query  the  result  of  relationally  joining  three  files  to  find  all  records 
containing  one  data  element  value  from  the  first  file  AND  one  data 
element  value  from  the  second  file  OR  one  data  element  value  from 
the  third  file  (i.e.,  A  AND  (B  OR  Q). 

2.  Query  the  same  joined  fUes  for  one  data  element  value  from  one  file 
OR  one  data  element  value  from  the  second  file  AND  one  data  element 
value  from  the  third  file  (i.e.,  A  OR  (B  AND  C)). 

3.  Query  the  joined  files  to  find  all  records  containing  one  data  element 
value  from  one  file  AND  one  data  element  value  from  the  second  file 
AND  NOT  one  data  element  value  from  the  third  file.  Print  or  display 
the  results  showing  data  elements  from  all  three  files  (i.e.,  A  AND 

(B  AND  NOT  O). 

4.  Query  the  same  joined  files  for  one  data  element  value  from  one  file  OR 
one  data  element  value  from  the  second  file  AND  NOT  one  data  element 
value  from  the  third  file.   Print  or  display  the  results  showing  multiple 
elements  from  aU  three  fHes  (i.e.,  A  OR  (B  AND  NOT  C)). 

Total 

Average  (TotaV4) 

5G.  Traps  command  or  statement  errors 

Enter  a  command  or  statement  that  contains  a  known  identifiable  error 
such  as  a  keyword  misspelling,  illegal  use  of  an  operator,  etc.,  to  see 
how  the  4GL  traps  and  informs  the  user  of  the  error. 

5H.  Facilitates  correction  of  errors 

Correct  the  error  from  the  previous  task  and  resubmit  the  command  to 
see  how  the  4GL  facilitates  the  correction  of  errors. 

51.  Compiles  code 

Compile  a  number  of  commands  or  statements.  (This  may  not  be  possible 
with  some  4GLs  that  run  only  in  interpretive  mode.  If  so,  rate 
functionality  and  ease  as  0). 

5 J.  Interprets  code 

Interpret  a  statement.  (This  may  not  be  possible  with  some  4GLs  that 
run  only  in  compiled  mode.  If  so,  rate  functionaUty  and  ease  as  0.) 
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5K.  Condition  logic 

1.  Execute  a  simple  condition  statement  with  your  query  or  report  request. 

2.  Execute  a  simple  condition  statement  with  altemative  true  and  false 
outcomes. 

3.  Execute  a  compound  condition  statement. 

4.  Execute  a  compound  condition  statement  that  produces  the  complement 
result  of  the  previous  task. 

Total 

Average  (Total/4) 

5L.  Looping  logic 

Execute  an  iterative  loop  structure  such  as  repeating  until  the  number 
of  records  retumed  equals  50.  Do  this  as  part  of  a  query  or  report, 
i.e.,  do  not  program. 

5M.  Intrinsic  date/time  functions 

1.  Display  the  values  of  intrinsic  date/time  functions  such  as  DATE,  DAY, 
MONTH,  DAY  OF  WEEK,  DATE  PLUS-OR-MINUS  A  TIME  PERIOD, 
WEEK,  QUARTER,  YEAR,  TODAY'S  DATE,  TIME. 

2.  Store  the  results  of  intrinsic  date/time  functions  in  a  database. 

Total 

Average  (TotaV2) 

5N.  Intrinsic  numeric  functions 

1.  Display  the  values  of  intrinsic  descriptive  data  functions  such  as 
MINIMUM,  MAXIMUM,  AVERAGE,  COUNT,  TOTAL,  SUBTOTAL. 

2.  Store  the  results  of  intrinsic  descriptive  data  functions  in  a  database. 

Total 

Average  (Total/2) 

50.  Statistical  functions 

1.  Perform  the  statistical  calculations  using  intrinsic  statistical  routines  if 
available.  Check  results  for  accuracy.  Repeat  task  for  each  function 
required.  (Add  other  functions  as  required  by  your  apphcation.) 

Standard  deviation 

Variance 
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Regressional  analysis  _ 

Correlation  _ 

Discriminant  analysis  _ 

Factor  analysis  _ 

Descriptive  statistics  _ 

2.  Store  the  statistical  results  generated  in  #1  to  a  database.  Repeat  for 
each  function  required.   (Add  other  functions  as  required.) 

Standard  deviation  _ 

Variance  _ 

Regressional  analysis  _ 

Correlation  _ 

Discriminant  analysis  _ 

Factor  analysis  _ 

Descriptive  statistics  _ 

Total  _ 

Average  (Total/#)  

5P.  Ability  to  process  missing  data 

1.  Create  or  use  a  file  with  missing  values  in  a  numeric  field.   Compute  the 
average  of  this  numeric  field  including  the  records  with  missing  values.  _ 

2.  Using  the  same  file,  average  the  numeric  field  excluding  the  records  with 
missing  values.  _ 

3.  Compare  the  two  results.  If  the  4GL  has  correctly  processed  missing  values 
the  average  excluding  the  missing  value  records  wiU  be  higher  than  the 
average  including  the  missing  value  records.  If  it  correctly  processed  the 
missing  values  score  a  three  for  fiinctionaUty.  If  it  did  not,  score  zero.  _ 

Total  _ 

Average  (Total/3)  _ 
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4.6  Graphics  Features 

The  goal  of  this  section  is  to  evaluate  the  graphics  features  that  are  incorporated  in  the  4GL.  Some 
4GLs  actually  link  to  separate  graphics  software  while  others  have  integrated  aU  of  the  graphics 
functions  with  the  database  and  language  fiinctionality  of  the  4GL.  You  need  to  evaluate  the  ease 
of  use  of  various  features,  how  much  effort,  rekeying,  etc.,  it  takes  to  generate  certain  kinds  of 
graphs.  You  also  need  to  evaluate  the  quality  of  the  graphs  generated  on  the  screen  and  in  a 
printed  version,  discounting  the  quality  of  the  printer  itself. 


6A.  Bar  graph 

Create  a  bar  graph  from  data  in  the  database.     (6A) 

6B.  Pie  chart 

Create  a  pie  chart  from  data  in  the  database.     (6B) 

6C.  Exploded  pie  chart 

Create  an  exploded  pie  chart  from  data  in  the  database.     (6C) 

6D.  Scatter  diagram 

Create  a  scatter  diagram  from  data  in  the  database.     (6D) 

6E.  Line  graph 

Create  a  line  graph  from  data  in  the  database.     (6E) 

6F.  Connected  point  plot 

Create  a  connected  point  plot  from  data  in  the  database.     (6F) 

6G.  Trend  line 

Create  a  scatter  diagram  that  incorporates  a  trend  line.     (6G) 

6H.  Curve  fitting 

Create  a  graph  that  requires  curve  fitting  or  smoothing.     (6H) 

61.  Automatic  scaling 

Create  one  graph  with  data  points  and  a  second  graph  with  the  data 

points  multiplied  by  1000  to  demonstrate  automatic  scaling.     (61) 

6J.  Logarithmic  scaling 

Create  a  graph  whose  data  points  would  require  logarithmic  scaUng.     (6J) 

6K.  Labels  and  titles 

Generate  field  labels  and  titles  for  any  graph  from  the  data  dictionary 

(do  not  reenter  labels  and  titles).     (6K) 


Functionality 


Performance/Ease  of  use 


RATING 
SCALE 


0-  none 

1-  some 

2-  most 

3-  all 


0-  not  done/impossible 

1-  poorly /hindrance 

2-  adequateiy/helpful 

3-  easily/very  helpful 


31 


6L.  Multiple  fonts 

Create  a  graph  which  utihzes  multiple  fonts  (one  font  for  titles,  another 

for  labels,  etc.)     (6L) 

6M.  Calculated  values 

Create  a  line  graph  with  one  line  representing  computed  values  (such 

as  a  ratio  of  two  fields  in  the  database).     (6M) 

6N.  Grid  option 

Add  a  grid  pattem  to  any  graph.     (6N) 

60.  Color  option 

Add  graph  colors  for  either  print  or  display.     (60) 

6P.  Hatch  pattern  options 

Change  the  hatch  pattems  for  any  bar  graph.     (6P) 

6Q.  Ability  to  process  missing  values 

Create  a  line  graph  representing  a  statistic  that  includes  processing 

missing  values.     (6Q) 


4.7  Security  Features 

These  tasks  test  some  of  the  basic  security  features  of  most  4GLs.  Most  of  these  features  are 
implemented  very  differently  from  product  to  product.  Some  4GLs  have  separate  workspaces 
identified  for  each  user  so  identification  and  password  protection  are  done  at  this  level.  Others 
require  identifiers  and  passwords  for  each  file  or  procedure.  Some  4GLs  have  various  levels  of 
security  allowing  quite  complex  security  while  others  provide  only  read/write  protection  for  files. 

These  tasks  were  not  designed  to  represent  ALL  of  the  security  tasks  that  could  be  tested.  They 
were  designed  to  test  the  more  common  kinds  of  security  that  is  generally  provided  by  4GLs.  You 
may  want  to  add  additional  tasks  if  additional  levels  of  security  are  required  for  your  application. 
For  example,  to  provide  a  minimum  test  for  encrypted  files,  you  may  want  to  run  an  analysis  of 
the  character  distribution  of  the  files  before  and  after  encryption.  You  may  also  want  to  tailor 
these  tasks  to  your  particular  application.  For  example,  the  section  on  command  security  could  be 
altered  to  test  protection  fi-om  use  of  specific  commands  or  programs  that  are  of  interest  for  your 
application. 

One  other  important  part  of  security  has  not  been  included  in  these  tasks,  backup  and  recovery. 
Since  backup  and  recovery  are  often  dependent  on  other  facilities  of  the  hardware  and  software 
environment  it  is  difficult  to  generalize  tasks  that  could  run  on  microprocessor  versions  of  the  4GL 
and  be  an  indication  of  mainframe  or  minicomputer  versions  of  the  same  4GL.  You  could  tum 
off  the  power  in  the  middle  of  mnning  a  query  to  analyze  what  happens  to  the  data  but  this  would 
not  necessarily  indicate  what  would  happen  with  the  same  4GL  in  a  different  operating  environment. 
Some  4GLs  running  on  mainframes  or  minis  wiU  mark  data  that  has  been  involved  in  a  crash, 
provide  rollback  facUities,  and  issue  wamings  that  there  has  been  a  crash.  There  are  few  that  offer 
this  protection  in  the  microcomputer  versions.  Networked  versions  of  the  4GL  also  offer  different 
facilities  than  stand-alone  micro  versions. 
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Since  this  area  cannot  be  tested  as  part  of  the  functional  benchmarks,  we  recommend  that  you  talk 
to  the  vendor  and  other  users  of  the  4GL  under  consideration,  to  gather  information  about  backup 
and  recovery  on  the  hardware  and  software  environment  you  require.  Actual  case  studies  from 
other  users  of  what  happened  when  the  power  went  off  or  when  the  disk  crashed  would  be 
invaluable. 


RATING 
SCALE 


Functionality 

0-  none 

1-  some 

2-  most 

3-  alI 


Performance/Ease  of  use 

0-  not  done/impossible 

1-  poorly/hindrance 

2-  adequateIy/helpful 

3-  easily/very  helpful 


7A.  Specify  user  access 

Grant  unlimited  file  rights  to  a  specified  user.  Access  the  file  as  the 
user.  Attempt  to  access  the  file  as  a  different  user  to  ensure  that  access 
is  denied. 


(7A) 


7B.  Specify  file  security 

1.  Limit  a  user's  file  rights  to  read  only  privileges.  Display  the  file. 

Attempt  to  add  a  record  to  see  if  the  request  is  denied.  Attempt  to  delete 
a  record;  to  edit  a  record. 


2.  Establish  write  privileges  for  a  user.  View  the  file.  Add  a  record. 
Edit  data  in  a  record.  Delete  a  record. 


3.  Deny  access  to  a  file  for  read  or  write  privileges  for  a  user.  Access 
the  system  as  this  user.  Attempt  to  read  or  query  the  file  to  verify  that 
this  is  not  possible.  Obtain  a  listing  or  access  a  screen  with  the  database 
files  listed  to  ensure  that  the  file  is  not  listed. 


Total 


Average  (Total/3)    (7B) 

7C.  Limit  command  or  procedure  usage 

1.  Deny  a  user's  rights  to  update  a  file.  Access  the  system  as  the  user. 

Attempt  to  change  or  modify  a  record  to  verify  that  this  is  not  possible.     

2.  Grant  a  user  the  right  to  retrieve  no  more  than  10  records  fi"om  any 
query.  Attempt  to  display  more  than  10  records  to  determine  that  this 

is  not  permissible.     


3.  Create  a  command  procedure,  subroutine  or  program  to  make  a  listing 
of  the  files  in  the  database.  Set  the  security  to  prevent  a  user  from 
running  this  procedure.  Access  the  system  as  this  user  and  attempt  to 
run  the  procedure  to  ensure  that  this  is  not  possible. 
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4.  Block  a  user  from  using  the  program  editor  to  change  any  program 
(restrict  the  use  of  the  EDIT  command).  Access  the  system  as  this 
user  and  attempt  to  edit  a  program  file. 

Total 

Average  (Total/4) 

7D.  Specify  record  access 

1.  Place  limitations  on  the  accessibility  of  certain  records  in  a  file.  Display 
the  file  to  make  sure  restricted  records  are  not  shown. 

2.  Join  the  file  with  limited  record  access  to  a  file  that  has  unlimited 
access.  Display  the  contents  of  the  joined  files  to  see  if  the  records 
remain  hidden.   (A  score  of  three  should  be  given  if  the  record 
remains  hidden,  zero  if  the  record  is  now  accessible.) 

Total 

Average  (Total/2) 

7E.  Specify  field  access 

1.  Limit  the  access  capabilities  of  one  field  in  a  file.  Display  the  contents 
of  the  file  to  make  certain  the  field  does  not  show.  If  possible,  list  the 
structure  or  dictionary  of  the  file  to  see  if  the  field  is  hidden, 

2.  Join  the  file  with  a  limited  field  to  a  file  that  has  unlimited  access. 
Display  the  contents  to  see  if  the  field  is  still  hidden.  (A  score  of  three 
should  be  given  if  the  field  remains  hidden,  zero  if  the  field  is  now 
accessible). 

3.  Grant  a  user  the  right  to  update  only  certain  fields  in  a  file  (i.e., 
some  fields  cannot  be  updated). 

Total 

Average  (Total/3) 

7F.  Encrypt/decrypt  files 

1.  Encrypt  a  data  file.  Attempt  to  view  the  file  (type  or  list  from  the 
operating  system)  without  decrypting  to  see  if  the  file  appears  readable. 

2.  Decrypt  the  data  file.  View  the  fUe  to  make  certain  the  data  is  readable. 
Compare  with  original  data  to  ensure  the  file  contents  were  not  altered. 

Total 

Average  (Total/2) 
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7G.  Encrypt/decrypt  data  dictionary 

1.  Encrypt  the  data  dictionary.  Attempt  to  view  the  data  dictionary  to  see 
if  it  appears  readable. 


2.  Decrypt  the  data  dictionary.  Compare  with  the  original  data  dictionary 
to  make  certain  it  is  unchanged. 

Total 

Average  (Total/2) 

7H.  Default  logon  procedure 

Create  a  procedure  or  program  that  is  automatically  executed  whenever 
the  4GL  is  started.  Have  the  logon  procedure  or  profile  tailor  the 
system  to  the  user's  requirements. 

71.  User  profile 

Define  a  user  profile  to  Umit  access  to  the  system,  (i.e.,  accessing  the 
4GL  requires  the  user  to  enter  a  user  ID  and/or  password  before  access 
to  the  system  is  granted.  Only  acceptable  ID's/passwords  are  granted 
user  privileges).  Note:  Some  4GLs  do  this  automatically,  some  have  to 
be  programmed. 

7J.  Define  system  administrator 

Establish  a  user  ID  and/or  password  for  a  user  or  users  that  grants  this 
user  the  right  to  establish  IDs,  passwords,  file  access,  etc.,  for  other 
users.  Evaluate  the  protection  for  this  user  ID  and  password.   (Are  the 
ID  and  password  encrypted?  Are  they  missing  from  the  default  listing 
of  users,  etc?) 

7K.  Feature  restrictions 

1.  Restrict  a  user,  based  on  user  ID  and/or  password,  to  only  one  part  of 
the  4GL  (e.g.,  the  report  writer).  Log  in  to  the  4GL  as  the  user  and 
attempt  to  access  other  parts  of  the  4GL  (e.g.,  screen  generator)  to 
determine  if  access  is  denied. 

2.  Change  a  user's  ID/password  without  having  to  redefine  the  restrictions 
imposed  on  the  user.  Log  in  to  the  4GL  with  the  new  ID/password  and 
make  certain  the  same  restrictions  are  still  valid. 

3.  Delete  the  user's  ID/password.  Attempt  to  log  in  to  the  4GL  with  the 
password  to  make  sure  access  is  now  denied. 

Total 

Average  (TotaV3) 
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7L.  Logon  audit  Hie 

Create  a  file  that  records  the  name  of  the  user,  the  time  and  date  the 

user  logged  on,  and  the  time  the  user  logged  off.  Display  the  user 

logon  file  and  verify  that  the  contents  are  correct.     (7L) 


4.8  Implementation  Features 

A  number  of  the  tasks  grouped  under  implementation  features,  such  as  rating  the  documentation, 
on-line  help,  diagnostics  and  messages,  are  not  really  rating  one  specific  task  but  rating  the  usage 
of  these  features  in  general.  Others  require  interfacing  to  other  languages  or  other  software. 
Whatever  interfaces  are  required  by  your  application  should  be  used  for  these  tasks. 


Functionality  Performance/Ease  of  use 

RATING         0-none  0-not  done/impossible 

SCALE  1-some  1-poorly/hindrance 

2-  most  2-adequately/heIpfuI 

3-  all  3-easily/very  helpful 


8A.  Interface  to  other  software  packages 

1.  Interface  to  an  extemal  software  package  (a  graphics  package,  a  statistical 
analysis  package,  etc.)  without  having  to  exit  the  4GL  with  a  "quit"  or 
"end"  type  of  command. 

2.  Move  data  from  the  4GL  to  the  other  package  and  display  the  resulting 
data.  (Did  the  data  dictionary  information  transfer  or  did  the  data  have 
to  be  redefined?  Were  the  data  types  correct,  i.e.  integers  stayed  integers, 
numeric  were  not  changed  to  character,  etc.) 


Total 


Average  (Total/2)    (8A) 

8B.  Interface  to  other  languages 

Execute  a  procedure  or  program  written  in  a  different  language  (Fortran, 

C,  another  4GL)  without  having  to  exit  the  4GL  with  a  "quit"  or  "end" 

type  of  command.   (The  execution  command  can  be  in  a  command  mode 

if  available  or  imbedded  in  a  4GL  program.)     (SB) 


8C.  Interface  to  operating  system 

Access  the  operating  system  (i.e.,  such  as  hsting  a  file  directory) 

without  having  to  exit  the  4GL  with  a  "quit"  or  "end"  type  of  command.     (8C) 

8D.  POSIX  compatibility 

Execute  the  4GL  in  a  POSIX  conforming  environment.  (See  [FIPS151], 

"Portable  Operating  System  Interface  for  Computer  Environments.")     (8D) 
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8E.  On-line  HELP 

Access  on-line  help  from  the  4GL  and  display  it  on  the  screen.  Also 
remember  if  you  used  the  on-line  help  for  other  benchmarking  tasks 

and  if  it  was  helpful.   (If  on-line  help  is  not  available,  rate  this  zero.)     (8E) 

8F.  Documentation 

Rate  the  helpfulness  of  the  documentation.  Was  it  up  to  date  with  the 
current  version?  Was  it  complete?  Did  the  4GL  do  what  the 

documentation  said  it  would  do?  Was  it  easy  to  understand?     (8F) 

8G.  Interface  to  external  editor/word  processor 

1.  Generate  commands,  programs,  or  file  descriptions,  etc.  in  a  word 
processor  or  different  program  editor.  Execute  the  code  to  ensure  the 

4GL  accepts  it.     

2.  Utilize  an  extemal  editor  or  word  processor  without  having  to  exit  the 

4GL  with  a  "quit"  or  "end"  type  of  command.     

Total     


Average  (Total/2)    (8G) 

8H.  Error  messages  and  warnings 

1.  Try  to  issue  a  query  or  report  request  with  an  illegal  field  name.  Was 

an  error  message  displayed  that  was  meaningful  and  helpful?     

2.  Based  on  experience  from  other  tasks,  are  warning  messages  displayed 
informing  the  user  that  a  query  or  other  task  might  take  a  long  time, 

that  a  certain  action  might  delete  a  file,  etc.?     

Total     


Average  (Total/2)  (8H) 

4.9  Application  Development 

There  are  certain  parts  of  the  4GLs  that  are  difficult  to  assess  in  a  piecemeal  fashion.  A  number 
of  these  are  part  of  the  process  of  actually  building  an  application.  The  pieces  of  an  application, 
such  as  screens,  reports,  etc.,  can  be  tested  as  we  have  listed  in  appropriate  sections  of  this  report. 
What  is  more  difficult  to  specify  in  individual  tasks  is  the  functionality  of  the  4GL  to  facilitate 
putting  these  pieces  together  to  build  an  application.  This  functionality  incorporates  the  robustness 
of  the  programming  or  procedural  language  with  the  ability  to  integrate  the  language  with  other 
segments  that  may  have  been  generated  by  the  nonprocedural  capabilities  of  the  4GL. 

Some  4GL  products  have  been  joined  with  CASE  products  or  other  software  design  tools.  This 
section  is  not  going  to  suggest  rating  these  capabilities  because  they  are  outside  of  the  scope  of 
most  4GLs.  This  section  also  does  not  attempt  to  retest  those  capabilities  that  have  been  Listed  in 
other  sections  of  this  report.  Instead  it  focuses  on  areas  that  cannot  be  tested  except  by  building 
an  application.  For  example,  some  4GLs  have  conditional  logic  that  can  be  used  in  report  or  query 
requests  (IF  a  value  =  x,  WHERE  a  code  is  less  than  some  number,  IF  the  report  does  not  exceed 
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a  record  limit,  etc.).  This  logic  and  syntax  is  sometimes  quite  different  from  logic  which  indicates 
branching  to  different  parts  of  a  program  (IF  something  occurs  then  execute  a  subroutine,  or 
process  a  different  section  of  the  program). 

We  suggest  that  you  design  and  build  a  very  small  application  that  would  be  essentially  a  small 
prototype  of  your  apphcation,  including  representative  files,  reports  and  queries.  The  application 
that  we  used,  described  below,  was  developed  in  an  average  of  four  hours  using  several  different 
4GL  products.  The  application  needs  to  be  developed  before  the  tasks  can  be  rated. 

EXAMPLE  APPLICATION 

Our  application  was  based  on  two  very  small  (only  10  records  each)  files.  One  file  had  an 
identifier  (employee  identification  number),  employee  name,  and  salary.  The  other  file  had  cost 
centers,  identifiers,  and  number  of  hours.  The  basic  idea  was  that  most  applications  would  have 
some  file  modifications  (changes  to  records,  adding  records,  or  deleting  records)  and  some  queries 
or  reports.  Our  application  required  posting  of  hours  to  modify  the  database,  and  reporting  the 
records  by  joining  the  files  and  multiplying  the  salary  times  the  number  of  hours.  We  reported 
the  records  by  employee  and  by  cost  center.  A  general  description  is  given  in  figure  2. 

Figure  3  gives  the  description  and  contents  of  the  two  files  that  were  used  for  the  application. 
These  files  were  kept  very  simple  since  these  tasks  were  not  designed  to  test  the  file  handling 
capabilities  of  the  4GL,  but  were  designed  to  test  the  procedural  language  capabilities.  Figure  4 
is  a  sample  of  the  kinds  of  screens  that  we  developed  and  figure  5  is  a  sample  of  one  of  the 
reports.  We  have  generalized  these  figures  to  represent  the  results  of  many  4GLs.  They  are  not 
actual  printouts  of  any  specific  4GL  product. 

Even  a  small  application  tests  some  of  the  procedural  language  capabilities  of  the  4GL.  The 
program  must  be  able  to  provide  the  three  basic  constructs  of  sequence,  iteration,  and  selection. 
The  program  can  incorporate  screens,  menus,  or  reports  that  come  from  the  nonprocedural  part  of 
the  4GL  but  the  screens,  menus,  and  reports  must  be  integrated  with  programming  logic  to  create 
a  stand-alone  application.  We  also  included  calculations  and  doUar  figures  with  floating  dollar 
signs  because  these  are  handled  differently  by  many  4GLs  and  are  also  typical  of  most  business 
applications. 

After  a  small  application  has  been  developed,  you  should  be  able  to  give  ratings  to  the  following 
tasks.  We  strongly  suggest  that  you  use  an  application  that  is  typical  of  your  large  application  or 
a  small  prototype  of  your  application.  The  requirements  should  be  kept  to  a  minimum,  however, 
to  prevent  these  tasks  from  being  overburdensome.  If  you  have  specialized  programming 
requirements,  then  we  suggest  that  you  add  tasks  to  this  area. 


Functionality 


Performance/Ease  of  use 


RATING 
SCALE 


0-  none 

1-  some 

2-  most 

3-  alI 


0-  not  done/impossible 

1-  pooriy/hindrance 

2-  adequately/helpful 

3-  easily/very  helpful 


9A.  Program  editor 

Rate  the  functionality  and  ease  of  use  of  the  4GL  program  editor. 


(9A) 
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Welcome  Screen 


File  Modification 


Report  or  Query 


Add  data 
Modify  data 
Delete  data 


Report  A 
Query  A 


Figure  2.  Test  Application 


TEST  APPLICATION 

FILES 

DESCRIPTION 

DESCRIPTION 

Filename : EMPLOYEE 

Filename : 

COSTCEN 

Field  1:  ID  -  4 

characters 

Field  1: 

ID  -  4  characters 

Field  2:   LASTNAME  -  10  characters 

Field  2: 

CEN  -  4 

characters 

Field  3:   FIRSTNAME  -  10  characters 

Field  3: 

HRS  -  3 

numbers  with 

Field  4:  SALARY 

-  5  numbers  with 

0 

decimal 

places 

2  decimal  places 

CONTENTS 

CONTENTS 

ID 

CEN 

HRS 

ID     LASTNAME  FIRSTNAME 

SALARY 

107 

0101 

0 

107  Madison 

Michele 

07.82 

210 

0202 

0 

210  Eisenhower 

Murph 

07.82 

312 

0101 

0 

312  Jones 

James 

09.68 

413 

0202 

0 

413  Truman 

Elizabeth 

09.68 

514 

0301 

0 

514  Wilson 

John 

11.85 

610 

0301 

0 

610  Swivit 

Rodney 

13.04 

715 

0101 

0 

715  Harrison 

Suzanne 

14.33 

807 

04P1 

0 

807  Roosevelt 

John 

17.17 

907 

0202 

0 

907  Garfield 

Elizabeth 

20  .42 

1015 

0101 

0 

1015  Harrison 

James 

24  .13 

Figure  3.  Application  Files 
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WELCOME 

TO  THE  4GL  TEST  SYSTEM! 

YOU  MAY:  1) 

Post  hours  for  employees 

2) 

Report  by  employee  or  cost  center 

3)  Exit  the  system 

MAKE  YOUR 

SELECTION  (enter  1,  2,  or  3): 

Figure  4,  Application  Screen 


TEST 

APPLICATION  REPORT 

SAMPLE 

REPORT  BY  COST  CENTER 

CEN 

ID 

LASTNAME  FIRSTNAME 

HRS 

DOLLARS 

101 

21 
71 
101 
TOTAL 

0 
5 
5 

FOR 

107 
COST 

Madison  Michele 
Eisenhower        Murph  20 
Harrison            Susanne  5 
Harrison            James  22 
CENTER  73 

26 

$156 
$71 
$530 
$962 

$203 

40 
65 
86 

23 

32 

202 

21 
90 
TOTAL 

0 
7 

FOR 

413 
COST 

Truman 
Eisenhower 
Garfield 
CENTER 

Elizabeth 
Murph  42 
Elizabeth  24 
101 

35 

$328 
$490 
$1157 

$338 

44 
08 
32 

80 

301 

61 

TOTAL 

0 

FOR 

514 
COST 

Wilson 

Swivit 
CENTER 

John 

Rodney  8 
60 

52 

$104 
$720 

$616 

32 
52 

20 

401 
TOTAL 

FOR 

715 
COST 

Harrison  Susanne 
CENTER  12 

12 

$171 

$171 

96 

96 

TOTAL 

246 

$3012 

03 

Figure  5.  Application  Report 
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9B.  Error  messages 

Rate  how  helpful  the  error  messages  were.  When  you  ran  the  appUcation, 
did  you  get  messages  that  merely  said  there  is  an  error  #xxx,  or  did  it 
explain  what  the  error  was  and  peihaps  contain  syntax  suggestions,  line 
numbers  of  the  errors,  etc? 

9C.  Branching  logic 

Were  you  able  to  move  to  different  sections  of  the  program  to  reflect 
choice  selection? 

9D.  Looping  logic 

Could  you  execute  a  process  multiple  times  using  iteration,  (e.g., 
multiple  reports,  multiple  records,  etc.)? 

9E.  Language  integration 

Was  the  programming  language  consistent  with  the  language  you  used 
for  conmiands  or  similar  to  the  language  of  menus  or  screens  provided 
in  the  nonprocedural  parts  of  the  4GL?  If  the  syntax  and  functionality 
are  consistent  across  the  4GL  rate  the  functionality  three. 

9F.  Ability  to  call  subroutines 

1.  Enter  a  block  of  statements  and  store  them  as  a  procedure  or  program 
(PROCEDURE  A).   Execute  PROCEDURE  A. 

2.  Enter  another  stored  procedure  (PROCEDURE  B),  one  statement  of 
which  executes  PROCEDURE  A.  Execute  PROCEDURE  B. 

Total 

Average  (Total/2) 

9G.  Ability  to  imbed  queries  in  procedures 

Create  a  report  procedure,  REPORT  A,  containing  a  conditional 
statement  to  compute  a  SUM  and  statements  to  query  a  data  file  for  a 
subset  of  the  file's  data. 


9H.  Ability  to  share  temporary  data 

Execute  a  procedure  that  extracts  a  temporary  subset  of  data  from  a  file 
and  makes  this  data  available  to  another  stored  procedure  within  the  4GL. 

9L  Ability  to  create  global  variables 

Define  global  memory  variables  (temporary  variables  which  are  accessible 
to  more  than  one  program). 

9J.  Ability  to' create  local  variables 

Define  local  memory  variables  (temporary  variables  which  are  accessible 
only  to  one  program). 

9K.  Ability  to  define  calculations  using  Held  values 

Define  a  memory  variable  that  is  a  result  of  a  calculation  between  two 
fields. 
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5.  4GL  BENCHMARKING  SUMMARY 


The  benchmarking  summary  sheet  can  be  used  for  the  final  step  in  evaluating  specific  products. 
The  requirements  weights  determined  how  important  a  feature  was  to  the  application.  The  scores 
on  the  functional  benchmark  tasks  indicated  how  well  the  4GL  exhibited  the  feature  and  how  easy 
it  was  to  perform  the  task  or  tasks  for  the  feature.  Taken  together  the  results  can  assist  a  decision 
maker  in  the  final  selection  of  a  4GL  product. 

The  weights  from  the  requirements  checklist  and  the  ratings  or  average  ratings  of  the  benchmark 
tasks  should  be  transferred  to  this  summary  sheet.  An  overall  rating  can  be  obtained  by  multiplying 
the  weights  times  the  sum  of  the  task  points  and  summing  these  points  for  each  product  evaluated. 
An  example  of  this  summary  process  is  given  below  using  part  of  data  management  as  an 
illustration. 


STEP  ONE  -  TRANSFER  REQUIREMENT  WEIGHTS 


A.  Basic  data  dictionary 

B.  Synonym  or  aliases 

C.  Headings,  titles,  or  labels 

D.  Comments  or  remarks 


E.  Date  formats 


F.  Time  formats 


Wt.  mult,  by 

Weight     FUNC   EASE  FUNC+EASE 

_5  

_1        

4       

_3  

4       

0     


STEP  TWO  -  TRANSFER  BENCHMARK  TASK  SCORES 


Wt.  mult,  by 

Weight      FUNC   EASE  FUNC+EASE 


A.  Basic  data  dictionary 

_5  

_3_ 

_2.5 

B.  Synonym  or  aliases 

_1  

_3_ 

_1_ 

C.  Headings,  titles,  or  labels 

4 

_0_ 

_0_ 

D.  Comments  or  remarks 

_3  

_3_ 

_3_ 

E.  Date  formats 

4 

_2_ 

_3_ 

F.  Time  formats 

0 

_0_ 

_0_ 
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STEP  THREE  -  MULTIPLY  WEIGHT  TIMES  SUM  OF  FUNC  AND  EASE  -  SUM  ALL 


Wt  mult,  by 

Weight     FUNC   EASE  FUNC+EASE 


A.  Basic  data  dictionary 

_5  

_3_ 

_2.5 

_27.5 

B,  Synonym  or  aliases 

_1  

_3_ 

_1_ 

4 

C.  Headings,  titles,  or  labels 

4 

_0_ 

_0_ 

0 

D.  Comments  or  remarks 

_3  

_3_ 

_3_ 

 18_ 

E.  Date  formats 

4 

_2_ 

_3_ 

20 

F.  Time  formats 

0 

_0_ 

_0_ 

0 

TOTAL  69.5 


4GL  SUMMARY  SHEET 

Wt.  mult,  by 

Weight     FUNC  EASE  FUNC+EASE 

L    OPERATIONAL  FEATURES 

A.  Installation  management         

B.  Ease  of  installing  on  multiple  platforms        

Z_  DATA  MANAGEMENT 

A.  Basic  data  dictionary         

B.  Synonym  or  aliases         

C.  Headings,  titles,  or  labels         

D.  Comments  or  remarks         

E.  Date  formats         

F.  Time  formats         

G.  Money  formats         

H.  Variable-length  fields         

I.  Capable  of  handling  #  chars  per 

field  
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Weight 


Wt.  mult,  by 
FUNC  EASE  FUNC+EASE 


J.   Capable  of  handling  #  fields 

per  record 

K.  Keyed  or  indexed  files 

L.  Data  structure  management 

M.  Ability  to  read,  query,  or  report 
from  ASCn  files 

N.  Ability  to  import  ASCII  files 

O.  Ability  to  export  ASCII  files 

P.  Ability  to  read,  query,  or  report 
from  formatted  files 

Q.  Ability  to  import  

formatted  files 

R.  Ability  to  export  

formatted  files 

S.  Ability  to  modify  files  while  loading 

T.  File  check  while  loading 

U.  File  modification 

3^  USER  INTERFACE 

A.  Capability  to  create  simple  menus 

B.  Capability  to  create  complex  menus 

C.  Default  data  entry  screens  provided 

D.  Capability  to  create  straight  text 
screens 

E.  Capability  to  create  display  screens 

F.  Data  checking  on  data  entry 

G.  Reverse  video  control 


44 


Weight 

H.  Blinking  data  entry  fields   

I.  Calculated  fields  display   

J.   Display  of  HELP  information  on  screen  

K.  Capability  to  generate  more  complex 

screens   

L.  Ability  to  print  screen  layout  document  

4^  REPORT  WRITING  FEATURES 

A.  Default  report  formats  provided  for 

paging,  dating,  and  headings   

B.  Default  report  formats  provided  for 
summary  functions   

C.  Default  report  selections  provided 

for  ascending  sorts   

D.  Default  report  selections  provided 

for  descending  sorts   

Complex  reporting  options  for: 

E.  Row  totals  and  subtotals   

F.  Percent  formats   

G.  Zero  suppression   

H.  Floating  dollar  signs   

I.  Comma  option  for  numeric  fields   

J.       User  specified  spacing   

K.      User  specified  page  breaks   

L.      Table  lookups   

M.  Headings   

N.  Footers   


Wt.  mult,  by 
FUNC   EASE  FUNCh-EASE 
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Wt.  mult,  by 

Weight      FUNC   EASE  FUNC+EASE 
O.      Font  selection       '   


P.  Odd  size  printouts 

Q.  Forms  printing 

R.  Suppressed  fields 

S.  Calculated  fields 


T.      Specific  field  placement 

U.      Output  device  independence  (i.e., 
user  could  select  screen,  printer, 
or  file) 

V.      Output  device  adjustments  (i.e., 
user  selects  screen,  printer,  or 
file  and  4GL  adjusts  for  page 
size,  screen  size,  etc.) 

5^  LANGUAGE  FEATURES 

A.  SQL  implementation 

B.  Simple  queries  on  one  file 

C.  AND  OR  logic  -  one  file 

D.  Relational  join 

E.  AND  OR  logic  -  two  files 

F.  Complex  Boolean  logic 

G.  Traps  command  or  statement  errors 

H.  Facilitates  correction  of  errors 
L  Compiles  code 

J.   Interprets  code 
K.  Condition  logic 
L.  Looping  logic 
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Weight 

M.  Intrinsic  date/time  functions 

(DATE,  DAY,  MONTH,  etc.)   

N.  Intrinsic  numeric  functions  (MINIMUM, 

MAXIMUM,  AVERAGE,  etc.)   

0.  Statistical  functions: 

Standard  deviation   

Variance   

Regressional  analysis   

Correlation   

Discriminant  analysis   

Factor  analysis   

Descriptive  statistics   

P.  Ability  to  process  missing  data   

6^  GRAPHICS  FEATURES 

A.  Ability  to  create  bar  graphs   

B.  Ability  to  create  pie  charts   

C.  Ability  to  create  exploded  pie  charts   

D.  Ability  to  create  scatter  diagrams   

E.  Ability  to  create  line  graphs   

F.  Ability  to  create  connected  point  plots   

G.  Ability  to  incorporate  trend  lines  in 

scatter  diagrams   

H.  Ability  to  curve  fit  or  smooth   

1.  Availability  of  automatic  scaling   

J.   Availability  of  logarithmic  scaling   


Wt.  mult,  by 
FUNC   EASE  FUNC+EASE 
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Weight 


Wt.  mult,  by 
FUNC  EASE  FUNC+EASE 


K.  Ability  to  generate  field  labels  and 
titles  from  the  data  dictionary 

L.  Ability  to  utilize  multiple  fonts 

M.  Ability  to  calculate  values  to  incor- 
porate in  the  graph 

N.  Grid  option 

0.  Color  option 

P.  Hatch  pattem  options 

Q.  Ability  to  process  missing  values 

7^  SECURITY  FEATURES 

A.  Ability  to  specify  user  access 

B.  Ability  to  specify  file  security 

C.  Ability  to  limit  command  or  procedure 
usage 

D.  Ability  to  specify  record  access 

E.  Ability  to  specify  field  access 
P.  Ability  to  encrypt/decrypt  files 

G.  Ability  to  encrypt/decrypt  data 
dictionary 

H.  Default  logon  procedure 

1.  User  profile 

J.  Ability  to  define  system  administrator 

K.  Peature  restrictions 

L.  Ability  to  create  a  user  logon  audit  file 
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Wt.  mult  by 

Weieht  FUNC  EASE  FUNC+EASE 

8^  IMPLEMENTATION  FEATURES 

A.  Interface  to  other  software  packages         

B.  Interface  to  other  languages         

C.  Interface  to  operating  system         

D.  POSIX  compatibility         

E.  Online  HELP         

F.  Documentation         

G.  Interface  to  external  editor/ 

word  processor         

H.  Error  messages  and  wamings         

9,    APPLICATION  DEVELOPMENT 

A.  Program  editor         

B.  Error  messages         

C.  Branching  logic         

D.  Looping  logic         

E.  Language  integration         

F.  Ability  to  call  subroutines         

G.  Ability  to  imbed  queries  in  procedures         

H.  Ability  to  share  temporary  data         

I.  Ability  to  create  global  variables         

J.   Ability  to  create  local  variables         

K.  Ability  to  define  calculations  using 

field  values         
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6.  ANALYSIS  AND  CONCLUSIONS 


After  the  complete  functional  benchmarking  process  has  been  repeated  for  each  4GL  product  under 
consideration,  the  total  numbers  can  be  compared.  A  higher  total  number  for  one  product  compared 
to  the  total  of  another  product  indicates  that  one  product  more  fully  met  the  requirements  of  an 
application  than  the  other. 

The  ratings  of  the  individual  tasks,  especially  for  the  highest  weighted  requirements,  should  also 
be  analyzed.  This  is  important  when  two  products  have  total  scores  that  are  very  close  to  each 
other.  Analyzing  individual  functional  areas  of  the  4GLs  reveal  the  strengths  and  weaknesses  of 
the  products  relative  to  the  requirements.  Because  certain  requirements  have  more  than  one  task, 
analysis  of  the  worksheets  for  each  task  may  also  be  informative.  An  analysis  of  variance  of 
individual  area  scores  may  tell  you  how  much  confidence  you  should  have  in  your  choice. 

Completion  of  the  tasks  and  the  evaluation  of  the  results  may  also  reveal  that  the  initial 
requirements  weights  should  be  adjusted.  Sometimes  requirements  that  seemed  unimportant  to  an 
application  are  discovered  to  be  more  important  than  originally  thought.  This  adjustment  of 
requirements  is  an  important  part  of  the  prototyping  process.  Completing  the  functional 
benchmarking  tasks  is  similar  to  completing  a  small  prototype.  A  more  thorough  understanding  of 
the  application's  requirements  is  gained  in  the  process. 

The  testing  methodology  and  the  set  of  functional  benchmark  tasks  provided  in  this  report  should 
facilitate  the  evaluation  of  4GL  products  in  relation  to  organizational  requirements.  Completing 
the  tasks  requires  some  effort  but  the  results  will  provide  a  more  thorough  understanding  of  the 
4GL  products  under  consideration.  This  should  allow  federal  managers  to  make  better  informed 
decisions  in  the  selection  and  procurement  process  for  a  4GL. 
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